Streaming IoT Data to Cloud Storage
good and practical
Some issues with the dataflow job creation step because of which the job is getting failed, the zone is getting assigned improperly.
I configured everything correctly but the file was not getting created in the cloud storage getting below error in data flow 2021-08-18T13:08:27.642700015ZWorkflow failed. Causes: There was a problem refreshing your credentials. Please check: 1. Dataflow API is enabled for your project. 2. Make sure both the Dataflow service account and the controller service account have sufficient permissions. If you are not specifying a controller service account, ensure the default Compute Engine service account [PROJECT_NUMBER]-email@example.com exists and has sufficient permissions. If you have deleted the default Compute Engine service account, you must specify a controller service account. For more information, see: https://cloud.google.com/dataflow/docs/concepts/security-and-permissions#security_and_permissions_for_pipelines_on_google_cloud_platform. , There is no cloudservices robot account for your project. Please ensure that the Dataflow API is enabled for your project.
8/14/21 - Great lab! Clear instructions, easy to follow along. When I encountered an error when I ran the job in Dataflow (in Run simulated device). It appeared that the error was a result that the worker location was us-central1-c on default, so I traced my steps back to Start a dataflow job and specified the workers zone=us-central1-a to be consistent with the Lab's project zone. The subsequent steps to Run the simulated devices returned what as expected and the simulation was successful through the Drain step (in Stop the Dataflow job).
Had to elevate to owner privileges for Dataproc to work. Instructions need to be updated to reflect it.