As we learn new ways to market and as a small business owner, you some times need to think out of the box. We have a wealth of data at our fingertip but unfortunately, not everything is straight forward and easy. Someone within my network suggested to use LinkedIn as the base for a newsletter that we're writing, so that we can bring more value to our customers and also be able to drive brand awareness more efficiently.
So the first step was to go ahead and export the LinkedIn network. Having a large network of a few thousands, it took a while to request an export file through the LinkedIn interface and I was excited when the file arrived to my inbox, so I rush to download it and go through my network to get an idea of the amount of information that I have.
Much to my dismay, I realized that the exported connections csv file did not have many emails. So I did some research and found out that you can opt out of being included in the export files through the LinkedIn Privacy and Security settings on this link right here:
https://www.linkedin.com/mypreferences/d/settings/email-address-visibility
So of course, first thing I did was check to see if I have "Only visible to me" checked. Then I set the "Allow connection to export emails" to OFF. From then on, I started looking into a few tools to be able to achieve my goal.
- Octopus CRM: https://octopuscrm.io/ which charges $39.99 US Dollars per month and did not work.
- SalesQL: https://salesql.com/ which charges $119 US Dollars per month and also did not work.
- Phantom Buster: https://phantombuster.com/ which charges $128 US Dollars per month which didn't make sense to me to try it out and have it not work.
And of course they have multiple tiers which did not get me what I wanted. If you have a smaller network, you can get away with the cheaper tiers to get an export of your network with their emails. But what do you do when your network is in the 4 figures or larger?
And I always think of the security implications, because these chrome extensions have access to the LinkedIn security tokens so they can run the exports or extraction in the background on their servers. So what happens when they are compromised or if they are malicious? Yep!
So I started diving deeper into the rabbit hole.
You can discover where the website is requesting traffic from, what URLs and what is it sending and receiving. This is the best tool for every web scraper out there! It's helped me so many times, I cannot begin to mention how useful it is.
So I opened a few pages on LinkedIn, especially the ones that display the profile's email address and discovered a graphQL call, which retrieves all the profile contact information.
And it looks something like this...
So from then on, I used a bit of Postman Interceptor, which connects to your Postman instance and catches all the calls being sent back and forth between your browser and the website API.
You can check it out here
Which is by the way, another great tool for Web Scrapers.
Anyways, I digress.
So after a bit of digging, and trial and error, turns out that the LinkedIn website uses cookies to track who you are, where you're coming from and a few other things, like your profile, theme, etc.
There's about 51 cookies installed by the LinkedIn website to validate that you're a logged in user and that the traffic is really coming from you and not a man in the middle. So kudos LinkedIn, high level security!
By the way you can check your cookies in your chrome browser by going into the Application tab in DevTools:
So I had a look at the values, trying to figure out which ones are used and which aren't in the GraphQL call, then I decided to use all of them.
After a bit of digging and trying, the web call in my C# application worked!
And this only took a few minutes by the way, because there's not enough time for personal fun projects, unfortunately!
Then I went ahead and started writing the code to do the following steps:
- Load the connections file which is in CSV format
- Go through the list and find which profiles did not have emails
- Call the GraphQL endpoint with the cookie and variables supplied
- Update the CSV file with the appropriate emails
- Back up the original file for safe keeping
And voila!
A full email extraction tool for my LinkedIn network, so about a few thousands of emails which we can use for our newsletter!
It has no monthly subscription fee!
It has no security implications, because I run it locally on my machine and I supply my own cookie information and no 3rd party entity has access to it!
So that's a win!
If anyone is interested in getting a copy, I'm making it available for Free.
Get in touch to get your copy or to hire us to work on your project.
Apologies for the UI it's a very basic tool, not intended for commercial use.
Disclaimer: Use at your own risk, LinkedIn has limits in place to safeguard against web scraping and email extraction so make sure you understand those limits based on your account type.
