I’m always interested in where my data is going. Who is using it? What data is being collected?

I use my phone a lot more then I use my (shiny new) computer so it made sense for me to see what data that is spilling. I also use an iPhone, so this can surely be ported to Android. I just don’t have the resources to do that.

What’s my goal?

  • I want to see what data my apps are sending and to where.
  • I want to be able to do this with tools I have on hand.

What do I need to achieve this goal?

First off. I do not need to buy an ASA on craigslist so I can monitor all the traffic. Yes, it’s fun. Yes it’s cool to see. But for my house, it’s overload and we have adequate heating. So, let’s not do that again. Okay, now that’s out of the way.

  • an iDevice with an app we want to see what it’s doing
  • a proxy
    • We will use Zap. Zap comes on Kali, or download it to your favorite platform.
  • the iDevice configured to use the proxy and accept the certificate

The Setup

We need to set up Zap to proxy our data as well as capture SSL traffic since most apps use https to make their calls.

Under the Tools –> Options –> Local Proxy let’s make sure the additional proxy is set to all ( and a different port then the localhost port. I have it set to 8082

additional proxies to

While in the options, let’s choose the Dynamic SSL Certificates and save that to the computer. We will need this in a few.

With this, we will be able to use apps and see the data being passed.

Device Setup

To get an iDevice to trust a cert we need to load it on the device and make some changes. The good thing is they don’t make this dead simple that it can happen by accident.

I will fire up Apple Configurator 2 then select File –> New Profile.

I will give it a name and an identifier.

Next I will give it the cert we made with Zap. This cert won’t be trusted, and that’s okay. We will take care of that in a bit.

Leave everything else as is and save it. It will save as a Configuration Profile file.

We are getting close to the fun part.

Transfer the file to the iDevice. I used AirDrop after emailing it to myself.

If you can’t AirDrop it, go for email or put it in Dropbox. When you get the file it should prompt you to install the profile.

Choose the Profile Downloaded

Install the Profile. It will warn you a few times. Read through the more details to see what’s happening. Always a good idea when following directions from the Internet.

You will get to the profile being installed. Click Done.


Now we need to trust the certificate. Just because it’s there, doesn’t mean we trust it. Navigate to the About Section, then scroll to the bottom to get the Certificate Trust Settings.

Enable the certificate that was presented and choose to continue.

Setting the Proxy

For all of this to work, we need to set the proxy for the network we are on.

Open the settings for the Wifi network. **This is specific to the network you are on. It will not proxy when on cellular or at your friends house or Starbucks.

It’s here that we set the proxy to the Zap server and port.

Click Save and we are ready to look at the data.

Viewing the data

Head on back to Zap and open up and app to see where the data is flowing.

One of the things I really like about Zap is that it groups the data by sites, so I can look to see what data is going where.

We can see we are getting https traffic and can decrypt the traffic just from woot here. We can see domains, requests, and responses.

Zap will let you manipulate this data to see what happens. That is for someone else’s blog post today.

Added bonus is seeing all the places that the app is sharing your data with and setting up a Pi-hole server to get rid of all that sharing your favorite weather app does.


We don’t always want our data going through the proxy. Some places are more finicky than others and will give weird results, so when done, always take the time to cleanup and undo the proxy settings and disable the trust in your certificate.

These can be turned back later when doing some more app reviews.

For a bit I was writing down the tools I had been working with and making. And then my blog blew up. Or more literally locked up and I lost the data because it was all on a dev machine that I didn’t care that much about.

I didn’t really stop working on things, but didn’t write much about it.

Then yesterday I had an idea. It wasn’t an original idea. It was really a how can I make that so I can use it and not need to install more software.

I came across this tool in a tweet. https://github.com/hakluke/hakcheckurl Written in Go, it checks on URLs, looks like it spiders and gets status codes for the URLs. Cool I thought. Go I thought.

Can I do it in python (I thought)? I played around. I looked around. I really didn’t want to rewrite a crawler. Lazy I know, but it’s my project and time.

New Direction

Sites have places they don’t want crawled. They put these places in a file in hopes that crawlers will respect this and not look there.

Most of these files/folders will be benign, style folders, images taken out of context, but some can help people looking for vulnerabilities out.

So, why not work out a way to take a look at them solo or in batches of sites?


Or as Gabe calls it, Skynet. It’s available at my github repo.

talking to a robot

So, what can it do?

Right now it’s pretty simple. Choose from one site or provide a list of sites and we will go check if they have a robots.txt file and log that data for review.

I’m hoping to add the ability to switch between http and https if one doesn’t show results soon for a site. The thought of piping the disallows to be followed and see what’s there has also crept into my mind.

Download it. Give it a spin. Give it a whirl. Please help me improve it.

When I was my sons age we heard about the horrors of Vietnam.  It was odd.  It was abstract.  I was a child.

I grew a bit and was then afraid of a nuclear war.  We were told there was a real chance.  99 red balloons.

Some lights shone through.  I remember Genesis and their promises with the Land of Confusion.  I remember Reagan yelling at Russia.

Shitty things were happening at home.  I spent some teenage years in the Midwest.

I went to see Neil Young and Crazy Horse at the onset of Desert Storm with my foster brothers and a teacher in my alternative high school.

Finish grad school and nurse a hangover from my best friends wedding as I make my way back to work on September 11.

Leave work that day and there are tanks on the corners around every Metro station.

Twenty Fucking Years later – we are still there.

Twenty Fucking Years later my kid is my age when I heard about the horrors of Vietnam.  They have to be as confused as I was.  Fuck, I am still confused.

Forty years later and I am still a pacifist.  There is always a better way.