Use git to control your config changes in /etc on snowflake servers

As a lightweight hobbyist and casual user of consumer electronics, I haven’t justified deploying any comprehensive server orchestration for my home machines or VPSs.  But as I tinker and experiment with server configurations, I don’t want to lose track of my thought process.  I also want to be able to back out of my changes without impossibly remembering what I’ve all done during the course of frantic experimentation.  The solution to this is as simple as could be.

This article makes these assumptions about the reader:

  • You know how to use git, or you can find that out on your own
  • If you’re modifying configs on your server then:
    • You know enough about your OS to infer how to use your package manager
    • You know when/whether to use sudo
    • You know how to edit files


  1. Install git
  2. cd /etc
  3. git init
  4. git status | grep -P ‘^\t’ | awk ‘{print $1}’ > .gitignore
    1. Note because this is a brand new init, you shouldn’t have any deleted: or modified: things.  But if you do either edit them out or add a grep -v pipe for them.
  5. Edit /etc/.gitignore
    1. Remove or comment any lines containing files you want to keep in version control.  For example, exim4/ bind9/ nginx/ might be good things to control.
  6. git add things that you want to control. Commit your changes as you wish.  I prefer to commit my changes as logical units.  So if I add a domain to my network, I might make one single commit to show my changes to nginx exim4 and bind9.


Use your discretion on how you want to use git.  You could add all of your /etc, but you probably don’t want to track password hashes nor some of the more unusual data or potentially dynamic changes that some services could keep in there.

Other Considerations

If I’m going to start making meaningful changes to something, I like to git add its directory and commit it before I make those changes.  That way I can have a known good point to revert to and I can also have a document of my thought process in making those changes.

Posted in Uncategorized | Leave a comment

Hackable Debian Router, Part 1: Philosophy


I am setting out to build a router from scratch, and document the journey in such a way that anybody can follow along.  But it’s not interesting enough to build a guide on IP Tables and a couple Network Cards, (albeit in this part we won’t get far beyond that).  I will work to satisfy my demands for a simple system using only the most trusted software packages that grants ultimate control (see “Goals” below) to a very paranoid home network owner.


I’m growing disappointed with the high cost, bland flexibility and stability of consumer network equipment.  At the same time I’m growing wary of the increasing number of network-connected devices that are accumulating in my home.  Botnets are growing in numbers and strength. Meanwhile our risk increases at a magnitude of average vulnerabilities per software package times average packages on a device times the number of devices on our home network!  You should take a perimeter walk around your home and see who’s connected, or if your router has a listing of DHCP clients, see how that list has grown beyond your expectations.  If you forgot to disable UPnP on your router, go take a look at how many things took it upon themselves to open up inward access from the Internet. Do you know why they’ve done this? How much do you trust those devices?  Do you even recognize all of them?  Do you have anything on your network that could be accessed without authentication (network-attached storage, printers, chromecast, security cameras)?

If you think I’m a paranoid lunatic, then you simply haven’t been following the news.

At the risk of dating this article into obsolescence, the references I’m linking above are all extremely recent to this writing, and to be honest quite cursory.  Anybody who follows tech news is fatigued by the nonstop onslaught of terrible mistakes.

This project isn’t going to solve those problems, but it will empower you to discover them with statistics & reporting, and shelter yourself at least while in your own home (assuming your phone is actually willing to stop routing traffic over the cellular network while on your WiFi).

Hardware Sustainability

Earlier in the section I mentioned the cost of equipment.  A functionally good router is going to be a couple hundred dollars.  Some people might think that’s a lot, and some might not.  But we also need to look at the hidden costs of these devices.

I have a TP-Link Archer C7 which actually has some pretty nice features and flexibility. Way cooler than anything I’ve ever had before, except for customized WRT firmwares, which brings me to my next point!  The importance of your firewall should not be overlooked.  Being able to run firewall software on commodity hardware is (in my opinion) way more sustainable.  When you buy any device, you’re actually buying a computer–and let’s be honest you’re buying a really crappy computer.  We’re talking weak CPUs, very little RAM, and you can’t update software packages without replacing the entire system by flashing firmware.


If you think you own a machine that has no bugs you are wrong.  They may not yet be discovered, but they are present. Meanwhile, corporations want to operate on their scheduled pace. If a flaw shows up in some library they depend on, you will not receive that patch until they get around to analyzing it, implementing it, building it, testing it and finally putting it to distribution. All of that comes after other business priorities.  You also have to be curious about which hardware models the business prioritizes, which takes me to one more question–how old is your router anyway?  You may never even be afforded the updates you need to keep your bank accounts safe.

In all fairness…

Non-commodity hardware is designed to do its very specific function, and to not be error prone.  Sure, you can misconfigure it or lose your password and get locked out or brick your unit altogether.  You can also go several years without upgrading your firmware and ultimately be assimilated into a botnet (or several).

Notes on surveillance

Yes everybody hates being under constant surveillance… and yet surveillance grows. At the same time we are filling our homes with devices that are at least as capable as mischievous gnomes. The aim of this project is gaining insight into what your machines are doing to the Internet on your behalf–or what the Internet is doing to you through your machines.  If you’re a sketch ball and worried about the data you collect being compromised or coerced from you, remember you have the power to configure your own retention policy.


Something like a decade ago I acquired too many roommates and and started having performance problems with an already aging linksys WRT.  I solved the problem by taking an incredibly more aging HP pavilion and set it up as my router gateway and firewall.  It was one of those goofy looking micro towers, but with a 50-watt power supply it seemed like a good choice for a computer that would be always on unattended.  I don’t remember a lot of details of that machine’s set up, but primarily it was barebones, with fail2ban and shorewall running on Debian (probably Debian 3.1).

Goals & Requirements

Keep in mind that these are the goals for this entire series project.  Especially with regards to features, we will not solve everything in this Part 1.

Software availability

  • Depend on only trusted packages available in and supported by official Linux distributions.
    • Depend on as few packages as possible.
  • Any assets developed for this project must be easily understood, and the distribution of them must be trustworthy.
  • Management and configuration of the solution must be easy to operate.


  • Management of various separated network zones
    • Ex. Network equipment, Personal Computers, Cameras, Misc IOT, Guest networks, Sandbox & Lab networks etc.
  • Collection and reporting of usage statistics–Dare I say surveillance?
    • TCP connection setups, durations, peers involved
    • UDP packet incidents
  • Hackability (As in customization, not vulnerable)
    • User-defined triggers
    • User-defined blocking triggers
      • See below for “Identify the nature of outbound traffic”

Concepts and Applications

If this project doesn’t produce a hackable customizable platform for truly owning your own network traffic, then it just isn’t worth doing (simple router solutions are already available).  Let’s describe some potential applications.

Origin management reporting

Collecting stats and building data sets and graphics that can be queried and filtered to expose who’s communicating through your network.  Leverage DNS queries to further identify peers and who owns them.

Aggressive outbound peer identification

Be able to enable a system of triggers that will disallow outbound connections to any IP address unless there was a recent DNS query seen associated with that address.

Origin blocking, ad blocking, or aggressive white listing

Use the firewall to keep AdBlock and uBlockOrigin extensions absolutely idle.

Conditional packet capture

If a traffic pattern is observed matching a particular set of conditions, begin recording it.


Before We Begin

I’m going to make some things clear. I am not a security expert! Not even close.  If you choose to roll out your own system based on the information detailed here, you do so at your own risk.

This project is not even close to my main focus, I pull it off the back burner when the following conditions are met: (1) I’m not at my full-time job. (2) I’m not doing household responsibilities. (3) I’m not enjoying the company of friends. (4) I’m not indulging in media. (5) I’m not tired. (6) I feel like working on this now.  This is a rare cross section of circumstances.  I expect severe consequences of this.

If you see me doing something dumb, please tell me.  I plan to revise this series as needed in order to harden the design, but not to keep it up with cutting edge technology.  If the technological landscape changes and I’m still interested, I would probably write a different edition of the series for that technology.  I’m also considering doing an edition for FreeBSD (with which I have no meaningful experience).


Expect revisions to this section until the series matures.

Part 1: Philosophy

You are here.  Explain the design goals and problem domain.

Part 2: Essential Functionality

We lay the ground work for building this firewall platform.  This remedial yet necessary process gets a machine up and running and able to route packets as a rudimentary gateway & firewall.  It will show how to describe separate network zones using several network interfaces and ip tables.  It will lay the foundation for the rest of our platform.

Part 3: Telemetry, Reporting, Observing

We figure out how to identify meaningful network events, and emit stats.  Stats can be shunted to some data-back end or processor.  We look some existing industry solutions to retain, query and present stats.  We look at techniques to discover odd traffic patterns, like a significant influx of traffic at a time when people should be sleeping or at work, or outliers like that one camera who just started going nuts.

Part 4: Triggering

We figure out how to get in the way of traffic matching a particular rule.  Once in the way, we can do whatever analysis is necessary and decide whether that traffic should be allowed to flow (or translated).

Part 5: Distribution

We go through the pain of taking our efforts and producing distributable packages that can easily be deployed and managed.

Part 0: Development Journal

Not exactly part of the series everyone will want to read, this article will be a heavily edited log of trials and tribulations throughout the development of this series.  It is meant to help me verify that I’m documenting everything in a complete fashion and in an appropriate chronological order.

Posted in Uncategorized | Leave a comment

Favorite gnome shell extensions

  • “No topleft hot corner” — Prevents the activities view when you mouse-over the  activities button
  • “Panel world clock” — Always set to GMT
  • “Pomodoro” — A simple pomodor timer
  • “Put windows” — super-arrow keys, super num keys to move windows around to different regions
  • “Workspace Grid” — (When it works,) lets you rearrange the workspace layout
  • “Mulitple monitor panels” — (When it works, ) displays window previews on both screens instead of just one.




Posted in Uncategorized | 1 Comment

Cheats to ignore files in git

I have this clump in my ~/.bashrc …

This provides the following commands:

git-ignore /some/filename/
git-unignore /some/file/name

## Git stuff, helpers
git config --global AndyO
git config --global core.excludesfile ~/.gitignore
git_ignore () 
 if grep -e "^$1\$" ~/.gitignore; then
 echo $1 already ignored;
 echo $1 >> ~/.gitignore;
git_unignore ()
 sed -e "/^$( sed -e 's/[]\/()$*.^|[]/\\&/g' <<< $1 )\$/d" ~/.gitignore -i
alias git-ignore=git_ignore
alias git-ignore-list='cat ~/.gitignore'
alias git-unignore=git_unignore
alias git-unignore-all='( > ~/.gitignore)'

If you desired, you could easily change the functions to loop over $@ to allow the user to provide more than one filename at a time.

Posted in Uncategorized | Tagged | Leave a comment

Practical use of LXC in Arch Linux in March of 2013

I’m writing this for myself in hopes that it benefits somebody else. The guides that I’ve found on LXC have been either unholy scatterbrained or just out of date.

Starting from scratch.

If you’re fairly new to arch as I am, keep in mind that if you hit your head on anything especially involving networking, reboot your system.  You may have had a system upgrade and had not yet rebooted.

Do not expect to understand LXC before you’ve thoroughly read this man page. . It’s short.

Get the stuff you need


pacman -Syy lxc bridge-utils

Set up your bridge


Edit /etc/network.d/bridge as



Now issue these commands

systemctl disable dhcpcd@eth0 # For boot time
systemctl stop dhcpcd@eth0  # For right now
systemctl enable netcfg@br0 # for boot time
systemctl start netcfg@br0 # for right now
systemctl enable dhcpcd@br0 # for boot time
systemctl start dhcpcd@br0 # for right now

For whatever reason this ^ isn’t working on startup for me.  I’ll revisit this.

Install a Debian Container

Get debootstrap

Get Yaourt


You need yaourt to install debootstrap


Follow the steps in this section here:


debootstrap lets you install a fresh debian system to a directory


yaourt -S debootstrap

Fix the create script

The create script doesn’t work.  Edit /usr/share/lxc/templates/lxc-debian to change



arch=’amd64′  # or whatever your architecture may be.

Now make it.

lxc-create -t debian -n mysupercoolcontainer

Wait! Stuff is still broke!

You can’t start your container until you weirdly remount your /, (per these comments

mount / –make-rprivate # Don’t sue me, I haven’t researched what this is, it might break everything.

Now start your container

lxc-start -n mysupercoolcontainer -d # -d makes it run as a daemon instead of forcing a console on your terminal.

Get in there

lxc-console -n mysupercoolcontainer # You’re in, you know what to do.


Posted in Uncategorized | Leave a comment

Favorite parts from “A Tour of Go” (in order)

If you happen to see this, bear with me I’m not done yet.

  1. Variable scope defined pre-exec to if-statements are available in else blocks.
  2. If-statements have a pre-exec section just like for-statments.
  3. As common, any portion of the for statement can be omitted (pre, while, post), but with one statement and no semicolon, it behave as a where.  Also for { … } is short-hand for an infinite loop.
  4. “Maps” in go are like dicts in python, but with a strictly typed key, but the key can apparently be any type! (In the tour example, try flipping the string and Vertex types in the map declaration as well as the definition, it works)
  5. You don’t have to dereference a pointer to use its value.


Go back and re-read these wierd things:

Posted in Uncategorized | Leave a comment

Shortcut for Ignoring Tracked Files in Git

Sometimes I make temporary changes to files that I need for the time being but never want to commit, while needing to commit other stuff. I can’t stash the changes because I need them constantly. Thanks to this article and its comments, I put together some aliases to help me out with this little dilemma.

alias git-ignore="git update-index --assume-unchanged "
alias git-ignore-list="git ls-files -v | grep -e '^h' | cut -d' ' -f2"
alias git-unignore="git update-index --no-assume-unchanged "
alias git-unignore-all="git update-index --no-assume-unchanged \`git ls-files -v | grep -e '^h' | cut -d' ' -f2\`"

Posted in Programming, Software | Leave a comment