Last week the EdgeRouter I use for my office did not come back after a reboot or perhaps I was too impatient, so I tried to reset it and that is when things went wrong.
Long story short, somehow I got the router into a state where it would not boot up anymore and the only way into it was the console port, for which I did not have a cable. Then I managed to find an old article which messed up my the routers USB drive so things took a turn for the worst. But fear not, it all worked out in the end, even though the only help I got from UBNT support was absolute zilch.
Here I outline some useful links for anyone else that may run into this issue, but mostly so I have it documented for next time ;). Before I get started here, the credit on everything here goes to others who documented this well on various locations.
Connecting to the console
In order to connect to the console, you will need a USB to RJ45 cable. I bought one off of Amazon which worked great right out of the box for me.
In order to connect (on a Mac), all you need to do is run:
> ls -ltr /dev/*usb* crw-rw-rw- 1 root wheel 21, 3 Jul 17 15:48 /dev/cu.usbserial-AI038TPF crw-rw-rw- 1 root wheel 21, 2 Jul 17 08:56 /dev/tty.usbserial-AI038TPF > screen /dev/tty.usbserial-AI038TPF 115200
There are a few links that were helpful here and I have them listed here, but I am going to outline what I had to do ultimately since I followed the first one and it messed me up more.
(ARCHIVED) EdgeRouter – Last Resort Recovery – DO NOT USE, Only as reference.
Recovering an unresponsive Ubiquiti EdgeRouter Lite router – DO NOT USE, Only as reference
EdgeRouter – Manual TFTP Recovery – Try this link first.
The third link above is perhaps the first thing you should try if you have not messed up your router bad enough, but I had to use the 4th and 5th links.
The EdgeRouter’s USB Drive
I kept reading about this, and could not believe that there was a usb drive in the edge router, but I guess ultimately that was a good design for when it goes bad and these routers used to have a history of the drives going bad.
There are three screws on the back of the router that you can open and the router comes apart (Note: you may void your warranty by doing this). Then the flash drive is right in your face.
Recovering the file system
Once the USB drive is unplugged, connect it to your PC/Mac and run the commands to recreate the drive. I found the easiest way was to use the mkeosdrive script provided in the last link above.
I ran the commands below, but if you read the GitHub site properly, there is a way to recreate the drive and include your backup in there as well.
# Get the path to the USB drive > sudo disk -l # Then run the command to create the drive > sudo ./mkeosdrive /dev/sdb ER-e100.v184.108.40.20699165.tar
Rebooting the router
Once the USB drive is ready, plug it back into your router, close things up. Then just wire it up and wait for it to boot. It should be back to normal.
I also found other links where folks talk about creating a backup of the USB drive in case something like this happens again, but what are the chances of that……right? 😉read more
[korey@localhost ~]$ brew install mongodbAt this point MongoDB is installed. To start it manually, first create the location where the DB will be stored (default is /data/db):
[korey@localhost ~]$ mkdir /data/db [korey@localhost ~]$ mongdNote that the user running mongod needs to have write access to the DB folder. The downside here is that the DB needs to be started manually each time and it will run as your userid. In order to automatically start the service, it is necessary to create a LaunchDaemon which will allow the service to start as soon as the computer starts.
1. The first step is to create a service account so the service does not run as root.
2. Next, is to create a proper location for the log and database location:
[korey@localhost ~]$ sudo mkdir -p /var/lib/mongodb [korey@localhost ~]$ sudo mkdir -p /var/log/mongo [korey@localhost ~]$ sudo chown -R _mongo:_mongo /var/lib/mongodb [korey@localhost ~]$ sudo chown -R _mongo:_mongo /var/log/mongo3. Now that we have an account and location, it is time to create the daemon plist file:
Store this file at: /Library/LaunchDaemons and name it: org.mongo.mongod.plist.
Label org.mongo.mongod ProgramArguments /usr/local/bin/mongod --dbpath /var/lib/mongodb/ --logpath /var/log/mongo/mongodb.log KeepAlive UserName _mongo GroupName _mongo
[korey@localhost ~]$ sudo launchctl load /Library/LaunchDaemons/org.mongo.mongod.plist [korey@localhost ~]$ sudo launchctl unload /Library/LaunchDaemons/org.mongo.mongod.plistread more
[korey@localhost ~]$ sudo dscl . -list /Users UniqueID _amavisd 83 _appleevents 55 _appowner 87 _appserver 79 _ard 67 _assetcache 235 _astris 245 _atsserver 97 _avbdeviced 229 _calendar 93 _ces 32 _clamav 82 _coreaudiod 202 _coremediaiod 236 _cvmsroot 212 ....The above command lists all the current users along with their UID. This is necessary so that we can pick an unused ID below 500 (UIDs above 500 are for normal users). You can run the same command with /Groups instead of /Users to get a list of groups. First, lets create a group for the users with the same name:
[korey@localhost ~]$ sudo dscl . -create /Groups/_mongo gid 300 [korey@localhost ~]$ sudo dscl . -create /Groups/_mongo RealName "Mongo DB Server Group" [korey@localhost ~]$ sudo dscl . -create /Groups/_mongo passwd "*"As you can see the group ID is set to 300, and the password is set to “*”. This is a special password not to allow logins as that group of user. I am not certain if this is necessary, but looking at other similar groups on OS X, it seems to be the right way to do this. Now, lets create the user and make sure that it will not show up as a user on the login screen:
[korey@localhost ~]$ sudo dscl . -create /Users/_mongo [korey@localhost ~]$ sudo dscl . -create /Users/_mongo uid 300 [korey@localhost ~]$ sudo dscl . -create /Users/_mongo gid 300 [korey@localhost ~]$ sudo dscl . -create /Users/_mongo NFSHomeDirectory /var/empty [korey@localhost ~]$ sudo dscl . -create /Users/_mongo UserShell /usr/bin/false [korey@localhost ~]$ sudo dscl . -create /Users/_mongo RealName "Mongo DB Server" [korey@localhost ~]$ sudo dscl . -create /Users/_mongo passwd "*"At this point, the service account is created, and its primary group set to the one we just created. Setting the shell and home folders are necessary to make sure that the account does not show up on the login screen and to ensure that even if someone does login as that user, they will not have access to anything. Once again, the account password here is set to “*” in order to not allow logins. If you look at /etc/passwd on your OS X machine, you’ll notice that most service accounts are listed in there, but the above account is not. I am not sure if this will be problematic over the long term, but for all intents and purposes, the service account works as expected. Naturally, I searched a good while before I came up with the above command set and here are some links that helped me:
/System/Library/Frameworks/CoreServices.framework/Frameworks/\ LaunchServices.framework/Support/lsregister -kill -r -domain local \ -domain system -domain userNote:
- This all has to be on one line, but I had to break it up to fit here with the back slash. So to execute, copy it to one line and remove the backslash.
- The location may be different if you are on an older version of OSX per the link above.
- Read the Google help on how to setup DKIM (DomainKeys Identified Mail) Signatures. Stop short of the last step to Start authentication, until you have finished all the DNS changes.
- Go to Unlock the Inbox and read about DKIM, DMARC (Domain-based Message Authentication, Reporting & Conformance), SPF (Sender Policy Framework) and ADSP (Author Domain Signing Policy). The combination of these will give you a pretty good level of security, although DKIM is the most important, followed by SPF.
- Create your DNS entries. For the sake of argument, assume the domain name is acme.com and remember that this was done on Site5, but Google has instructions for many popular providers.
- Login to you site’s cpanel
- Go to the advanced DNS editor.
- Enter the following four entries:
- Wait for a few hours so DNS propagates
- Go to KLOTH.NET and verify the settings with the Site5 DNS server
- Start authentication.
- As a final step, send an email from your domain to mailtest at unlocktheinbox dot com and you will receive a report on whether things are working or not.
Record Type: TXT Record Name: google._domainkey TTL: 3600 IP Address: v=DKIM1; k=rsa; p=MIGfMA0GCS...This is the DKIM record. The IP Address above is really the value and you should get it from Google when you generate you domain key. The Record name should be the same regardless of your domain name, just make sure not to end it in a dot, so it appends your domain to it. The end result for the name should be google._domainkey.acme.com.
Record Type: TXT Record Name: acme.com. TTL: 3600 IP Address: v=spf1 include:_spf.google.com ~allThis is the SPF record and not how we put in our domain name and ended it with a dot. The Site5 DNS editor does not allow for @, per Google’s instructions, so this is the only way to get around this.
Record Type: TXT Record Name: _adsp._domainkey TTL: 14400 IP Address: dkim=all;This is the ADSP record and again the name does not end with a dot so the domain gets appended to it just like our DKIM record.
Record Type: TXT Record Name: _dmarc TTL: 14400 IP Address: v=DMARC1; p=quarantine; adkim=s; aspf=s; \ rua=mailto:firstname.lastname@example.org; ruf=mailto:email@example.com; \ pct=100This was the trickiest one only because the Site5 DNS editor replaces the @ in the email with your domain name. What I found out is that if I pasted the value just like above, it saved fine, but it would not display right when I tried to edit it. The email addresses are not necessary unless you want to receive a report when emails are spoofed from your domain. Once again, the name here does not end in dot so that the domain will get appended to it and most importantly, the backslashes are just used as line breaks, but are not part of the actual string you need to input.
Domain: google._domainkey.acme.com Server: dns.site5.com Query: TXT (text)This should return the value you entered above once DNS has been updated. You can repeat the same process for the other entries to make sure as well.
- Add Google’s public DNS servers as static DNS servers for the DHCP server. Note that since my main router is using 192.168.1.x, I put this router on 192.168.11.x to make sure there is no complications.
- Configure the wireless access point on the router so that it does not conflict with your main wirless connection (i.e. give it a different name and use a different channel to be extra safe).
- Enable and configure the PPTP client on the router to connect to your favorite VPN location. You can get a list of the VPN location for Witopia here. For the Server IP or DNS Name I put in the IP address of the vpn server I wanted to connect to (e.g. pptp.chicago.witopia.net).
- Add a startup script to the router to configure it to use the VPN connection properly.
The script below will wait until VPN is connected and then update the router’s routing appropriately. Note that 192.168.1.1 is the internal IP of my main router, not the VPN router which is 192.168.11.1.
echo "echo \"Startup Config started\" >> /tmp/mylog.txt" > /tmp/startupConfig.sh echo PPTPSERVER=$(/usr/sbin/nvram get pptpd_client_srvip) >> /tmp/startupConfig.sh echo PPTPGWY=192.168.1.1 >> /tmp/startupConfig.sh echo "/sbin/route add -host \$PPTPSERVER gw \$PPTPGWY" >> /tmp/startupConfig.sh echo "#/sbin/route del default" >> /tmp/startupConfig.sh echo "/sbin/route add default gw \$PPTPGWY metric 100" >> /tmp/startupConfig.sh echo "/sbin/route add default dev ppp0" >> /tmp/startupConfig.sh echo "/sbin/route del default" >> /tmp/startupConfig.sh echo "/sbin/route del default" >> /tmp/startupConfig.sh echo iptables -t nat -A POSTROUTING -o ppp0 -j MASQUERADE >> /tmp/startupConfig.sh echo "ifconfig ppp0 > /dev/null" > /tmp/whileLoop.sh echo "RC=\$?" >> /tmp/whileLoop.sh echo "echo \"Checking ppp0: \$RC\" >> /tmp/mylog.txt" >> /tmp/whileLoop.sh echo "while [ \$RC -ne 0 ]; do" >> /tmp/whileLoop.sh echo " sleep 5" >> /tmp/whileLoop.sh echo " ifconfig ppp0 > /dev/null" >> /tmp/whileLoop.sh echo " RC=\$?" >> /tmp/whileLoop.sh echo " echo \"Checking ppp0: \$RC\" >> /tmp/mylog.txt" >> /tmp/whileLoop.sh echo "done" >> /tmp/whileLoop.sh echo "echo \"Running startupConfig.sh\" >> /tmp/mylog.txt" >> /tmp/whileLoop.sh echo "ifconfig ppp0 >> /tmp/mylog.txt" >> /tmp/whileLoop.sh echo "sh /tmp/startupConfig.sh" >> /tmp/whileLoop.sh sh /tmp/whileLoop.sh &
[korey@localhost ~]$ yum install mysql-server mysqlThe next thing is start MySQL:
[korey@localhost ~]$ service mysqld startOnce you start the service, it will give you some instructions for having it start automatically on reboot, and how to secure it. Finally, secure MySQL, by setting a password for the root user and removing the anonymous user. Just make sure you use the same password for the first two statements below, otherwise you’ll end up scratching your head as to why you cannot login sometimes.
[korey@localhost ~]$ mysql -u root
mysql> SET PASSWORD FOR 'root'@'localhost' = PASSWORD('****'); mysql> SET PASSWORD FOR 'root'@'localhost.localdomain' = PASSWORD('****'); mysql> DROP USER ''@'localhost.localdomain'; mysql> DROP USER ''@'localhost';read more
sudo vi /etc/pathshere is what mine looks like:
/usr/local/bin /usr/bin /bin /usr/sbin /sbinread more
ssh -o PreferredAuthentications="password" myUsername@myServerAddressread more