Firebase + Bigquery to search 1.4 Billion leaked passwords

September 04, 2019 in #firebase #bigquery #password | | | Share on Google+

Quick setup of powerful search tools

In this short demo, I will guide you how to get the passwords database loaded to Google Bigquery, and build an interface using Firebase for users to search for their own patterns.

Goal

User can search for a potential credential leak (email + password) in a simple online web page.

Functionalities

  • Data comes from a compiled dump from several past hacks. This post will show you how to load it to a BigQuery database
  • Firebase hosting provides a front to enter a search pattern
  • Firebase functions forwards the query to google BigQuery, and leaves a trace in Firebase Database
  • Firebase Database provides the administrator a way to see the user search history
  • Firebase auth (email + password) permits to link users and search terms

App workflow

1) User signs in using email + password, so we can map later his email and search pattern

2) When a user is logged in, it has access to the search bar.
Ps: for cost reason, I had to disable the search function for new users, as each BigQuery search costs 0.5$. Please ask me to unlock if you want to try the app.

3) The search query is sent to backend, which forward to BigQuery. Then the results, limited to 100 records, are displayed to user.

4) Backend records email + search + timestamp in the firebase database

Technical setup tips

Firebase project: first create a new Firebase project "gothacked". This will create a new google cloud project with the same name, which will store the Firebase infra

BigQuery: in my google cloud console, I selected the project "gothacked", and loaded the data using this guide. I ended up with a dataset passwords_from_torrent:

Where I could run local queries:

Credentials: because Firebase and BigQuery live in the same google project, Firebase backend functions don't need any additional credentials to access the data. Same for recording search history in the Firebase database

Code: I used some very basic JavaScript + html + css + bootstraps to code the frontend

CI-CD is so straight forward: you just have to tell to which Firebase project you want to deploy with firebase use $project, then deploy to cloud with firebase deploy

Conclusion

This side project taught me a lot about Firebase front/backend/database interactions.

I did not expect the ~0.5$ query cost on BigQuery, so this can't be open-sourced to many users. However, storing the data is inexpensive.

Other websites give you opportunities to check if your passwords have been revealed:

Next steps:

Thank you for reading :-) See you in the next post! Greg

September 04, 2019 in #firebase #bigquery #password | | | Share on Google+