Contributing to NiFi-11608

Today I finished my second Pull Request on the Apache NiFi Project for JIRA NiFi 11608. This JIRA describes an issue with PutBigQuery proccessor evaluating expression language inside the processor’s properties for the Dataset and Tablename. I came into the issue when a community user posted a question asking why it was working in PutBigQueryBatch but not in PutBigQuery.

You can find the PR here: NiFi-11608

Here are some useful commands I needed while building:

mvn clean install -T2C -am -pl :nifi-gcp-nar
mvn clean install -T2C -am -pl :nifi-gcp-nar -DskipTests

With a successful Build:

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for nifi 2.0.0-SNAPSHOT:
[INFO] 
[INFO] nifi ............................................... SUCCESS [  0.764 s]
[INFO] nifi-commons ....................................... SUCCESS [  0.047 s]
[INFO] nifi-api ........................................... SUCCESS [  6.779 s]
[INFO] nifi-framework-api ................................. SUCCESS [  9.400 s]
[INFO] nifi-property-utils ................................ SUCCESS [  5.337 s]
[INFO] nifi-utils ......................................... SUCCESS [ 15.698 s]
[INFO] nifi-data-provenance-utils ......................... SUCCESS [  1.718 s]
[INFO] nifi-deprecation-log ............................... SUCCESS [  6.319 s]
[INFO] nifi-parameter ..................................... SUCCESS [  8.521 s]
[INFO] nifi-uuid5 ......................................... SUCCESS [  2.476 s]
[INFO] nifi-expression-language ........................... SUCCESS [  9.995 s]
[INFO] nifi-xml-processing ................................ SUCCESS [ 14.941 s]
[INFO] nifi-json-utils .................................... SUCCESS [  8.432 s]
[INFO] nifi-properties .................................... SUCCESS [  8.851 s]
[INFO] nifi-security-utils-api ............................ SUCCESS [  4.559 s]
[INFO] nifi-security-utils ................................ SUCCESS [ 12.868 s]
[INFO] nifi-record ........................................ SUCCESS [  8.626 s]
[INFO] nifi-security-kerberos-api ......................... SUCCESS [  2.031 s]
[INFO] nifi-security-socket-ssl ........................... SUCCESS [  3.519 s]
[INFO] nifi-security-ssl .................................. SUCCESS [  5.460 s]
[INFO] c2 ................................................. SUCCESS [  0.049 s]
[INFO] c2-protocol ........................................ SUCCESS [  0.057 s]
[INFO] c2-protocol-component-api .......................... SUCCESS [  3.423 s]
[INFO] nifi-nar-bundles ................................... SUCCESS [  0.426 s]
[INFO] nifi-framework-bundle .............................. SUCCESS [  0.053 s]
[INFO] nifi-framework ..................................... SUCCESS [  0.022 s]
[INFO] nifi-client-dto .................................... SUCCESS [  6.467 s]
[INFO] nifi-site-to-site-client ........................... SUCCESS [ 22.009 s]
[INFO] nifi-web-client-api ................................ SUCCESS [  1.726 s]
[INFO] nifi-server-api .................................... SUCCESS [  0.929 s]
[INFO] nifi-nar-utils ..................................... SUCCESS [  3.693 s]
[INFO] nifi-py4j-bundle ................................... SUCCESS [  0.044 s]
[INFO] nifi-python-framework-api .......................... SUCCESS [  1.981 s]
[INFO] nifi-manifest ...................................... SUCCESS [  0.050 s]
[INFO] nifi-extension-manifest-model ...................... SUCCESS [  2.625 s]
[INFO] nifi-extension-manifest-parser ..................... SUCCESS [  4.428 s]
[INFO] nifi-runtime-manifest-core ......................... SUCCESS [  3.493 s]
[INFO] nifi-framework-nar-utils ........................... SUCCESS [  2.533 s]
[INFO] nifi-mock-authorizer ............................... SUCCESS [  0.890 s]
[INFO] nifi-framework-authorization ....................... SUCCESS [  2.488 s]
[INFO] nifi-framework-core-api ............................ SUCCESS [  3.129 s]
[INFO] nifi-mock .......................................... SUCCESS [  2.370 s]
[INFO] nifi-standard-services ............................. SUCCESS [  0.037 s]
[INFO] nifi-record-serialization-service-api .............. SUCCESS [  2.108 s]
[INFO] nifi-schema-registry-service-api ................... SUCCESS [  1.676 s]
[INFO] nifi-extension-utils ............................... SUCCESS [  0.044 s]
[INFO] nifi-record-utils .................................. SUCCESS [  0.026 s]
[INFO] nifi-standard-record-utils ......................... SUCCESS [  3.707 s]
[INFO] nifi-schema-inference-utils ........................ SUCCESS [  1.380 s]
[INFO] nifi-json-record-utils ............................. SUCCESS [  1.080 s]
[INFO] nifi-jetty-bundle .................................. SUCCESS [  0.610 s]
[INFO] nifi-kerberos-credentials-service-api .............. SUCCESS [  1.404 s]
[INFO] nifi-kerberos-user-service-api ..................... SUCCESS [  2.386 s]
[INFO] nifi-distributed-cache-client-service-api .......... SUCCESS [  2.366 s]
[INFO] nifi-ssl-context-service-api ....................... SUCCESS [  1.798 s]
[INFO] nifi-http-context-map-api .......................... SUCCESS [  1.404 s]
[INFO] nifi-dbcp-service-api .............................. SUCCESS [  1.252 s]
[INFO] nifi-hbase-client-service-api ...................... SUCCESS [  2.881 s]
[INFO] nifi-oauth2-provider-api ........................... SUCCESS [  5.617 s]
[INFO] nifi-rules-engine-service-api ...................... SUCCESS [  2.989 s]
[INFO] nifi-lookup-service-api ............................ SUCCESS [  2.089 s]
[INFO] nifi-proxy-configuration-api ....................... SUCCESS [  2.717 s]
[INFO] nifi-record-sink-api ............................... SUCCESS [  1.844 s]
[INFO] nifi-web-client-provider-bundle .................... SUCCESS [  0.030 s]
[INFO] nifi-web-client-provider-api ....................... SUCCESS [  1.877 s]
[INFO] nifi-key-service-api ............................... SUCCESS [  2.312 s]
[INFO] nifi-standard-services-api-nar ..................... SUCCESS [  0.262 s]
[INFO] nifi-avro-record-utils ............................. SUCCESS [  3.744 s]
[INFO] nifi-mock-record-utils ............................. SUCCESS [  2.034 s]
[INFO] nifi-listed-entity ................................. SUCCESS [  2.504 s]
[INFO] nifi-syslog-utils .................................. SUCCESS [  2.623 s]
[INFO] nifi-record-serialization-services-bundle .......... SUCCESS [  0.029 s]
[INFO] nifi-record-serialization-services ................. SUCCESS [  6.567 s]
[INFO] nifi-gcp-bundle .................................... SUCCESS [  0.046 s]
[INFO] nifi-gcp-services-api .............................. SUCCESS [  3.218 s]
[INFO] nifi-gcp-services-api-nar .......................... SUCCESS [  0.281 s]
[INFO] nifi-conflict-resolution ........................... SUCCESS [  2.504 s]
[INFO] nifi-gcp-processors ................................ SUCCESS [  7.915 s]
[INFO] nifi-gcp-parameter-providers ....................... SUCCESS [  4.214 s]
[INFO] nifi-gcp-nar ....................................... SUCCESS [  1.291 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:18 min (Wall Clock)
[INFO] Finished at: 2023-05-31T07:20:01-04:00
[INFO] ------------------------------------------------------------------------

I was able to copy the new nifi-gcp-nar straight into a local nifi on my desktop and test it directly:

cp .m2/repository/org/apache/nifi/nifi-gcp-nar/2.0.0-SNAPSHOT/nifi-gcp-nar-2.0.0-SNAPSHOT.nar /Users/steven.matison/nifi-1.21.0/lib/

Wow! I have to say, this new drop in nar functionality they added to NiFi is super helpful. Way to go to the NiFi Team for this big win!!

With the new nar added to the nifi lib I can see the original processor and my new processor:

Watch out! PutBigQueryBatch and PutBiqQueryStreaming are going away in nifi 2.0, so getting the PutBigQuery processor working was a much needed contribution.

Now I can begin to test the new processor side by side with the old processor:

Contributing to the open source project NiFi is very important to me here @ Cloudera. I am a firm believer in the open source community and look for chances to be helpful in this space. Watch for more to come as I work on more NiFi projects and document them here in my blog.