Disable InsecureRequestWarning When Using Python Requests Module

DISCLAIMER: You should NOT send any sensitive traffic to untrusted hosts on the Internet. This is method should only be used for troubleshooting or if you have independently verified the identify of the server you are connecting to.

The Python Requests module is a pretty cool and easy way to establish HTTP/HTTPS connections. However, if you ever try to connect to a server using HTTPS and the certificate is not trusted, you will probably get an error that looks something like this:

>>> import requests
>>> r = requests.get('https://fullyqualifiedurl.com')
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 70, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 56, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 497, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)

Before continuing on, I want to reiterate the disclaimer at the top of the page:

DISCLAIMER: You should NOT send any sensitive traffic to untrusted hosts on the Internet. This is method should only be used for troubleshooting or if you have independently verified the identify of the server you are connecting to.

Now that that's taken care of, the documentation for Requests says to pass the verify=False parameter when calling the requests.get method. Let's try that out:

>>> import requests
>>> r = requests.get('https://fullyqualifiedurl.com', verify=False)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:843: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)

Now we get a different error that has a link to documentation for urllib3. Since requests uses urllib3 under the hood, you need to disable certificate checking with urllib3 as well:

>>> import requests
>>> import urllib3
>>> urllib3.disable_warnings()
>>> r = requests.get('https://fullyqualifiedurl.com', verify=False)
>>> r.status_code
200

If you're still getting an error, it could be because your system is using the version of urllib3 that is bundled with requests. This should get things all squared away for you:

>>> import requests
>>> from requests.packages import urllib3
>>> urllib3.disable_warnings()
>>> r = requests.get('https://fullyqualifiedurl.com', verify=False)
>>> r.status_code
200

I think what python is doing behind the scenes is that if urllib3 is already installed, requests will use that version. But, if urllib3 is not already installed, requests will use its bundled version of urllib3. If you are writing code that might be used on different computers and you aren't sure what is installed, you can use this try/except block to import whichever version of urllib3 is applicable:

try:
    import urllib3
except ImportError:
    from requests.packages import urllib3

Disable Hibernate in Windows

Hibernate is a neat feature that is especially useful on laptops. Hibernate is different than sleep/standby in that with sleep/standby, the system shuts down most of the hardware, including the hard drive, but keeps the RAM powered to save the current state. This will (very) slowly drain the battery, but is extremely fast to resume. Hibernate saves the RAM state to hard disk and powers everything off. This is slower to resume than sleep/standby, but won't drain the battery and is still faster to resume than a full shutdown.

By default on Windows systems, hibernate is enabled. This includes servers, which should probably never enter hibernate state. The problem with hibernate, especially on older servers, is that the system reserves hard disk space large enough to store the contents of the RAM in the C:\hiberfil.sys file, which is a protected system file. If you have a small hard drive and a bunch of RAM, this can very quickly eat up a lot of valuable disk space. The only way to get this space back is to disable hibernate, which in most cases for servers, is a best practice anyway.

The easiest way to disable hibernate is to open a command prompt (cmd.exe) as administrator and run this command:

powercfg.exe -H off

Install ifconfig in RHEL or CentOS

Depending on the OS version and which configuration you selected during installation, you might discover that ifconfig is not installed by default in your new RHEL or CentOS build. Unfortunately, searching yum might not tell you the name of the package you need to install. I'll save you some looking, the package you need is called "net-tools."

You can install net-tools with this command:

# yum -y install net-tools

List All Domain Controllers for a Given Domain Using PowerShell

PowerShell is an awesome way to gather a bunch of useful information about the system you're on or the domains you're connected to.

This useful command will list all of the Domain Controllers for the given domain. Make sure to replace <domain> with your domain.

> Get-ADDomainController -Filter * -server <domain> | Select-Object name, domain

Use PowerShell To Delete Files Older Than

So you have a machine with a bunch of old files that are just taking up space and you want to quickly and easily clear them all out? PowerShell to the rescue. This command will delete all files in the current directory that were created more than 30 days ago. You can change the number of days based on your needs.

> Get-ChildItem | Where-Object{$_.CreationTime –lt (Get-Date).AddDays(-30)} | Remove-Item

Want to clear out the current directory AND all subdirectories? Add the -Recurse flag after Get-ChildItem.

> Get-ChildItem -Recurse | Where-Object{$_.CreationTime –lt (Get-Date).AddDays(-30)} | Remove-Item

Want to suppress any "are you sure" prompts? Add the -Recurse flag after Remove-Item.

> Get-ChildItem -Recurse | Where-Object{$_.CreationTime –lt (Get-Date).AddDays(-30)} | Remove-Item -Recurse

Determine PowerShell Version

PowerShell Cmdlets and syntax will sometimes vary from version to version. Its nice to have an easy way to check what version of PowerShell is installed on a system.There are a couple of ways to to this:

> $PSVersionTable.PSVersion

Major  Minor  Build  Revision
-----  -----  -----  --------
5      0      10586  122

Alternatively, you can also use the Get-Host command:

> (get-host).version

Major  Minor  Build  Revision
-----  -----  -----  --------
5      0      10586  122

Write ISO to USB drive in OS X

OS X systems natively support writing disk images to USB drives, however they often need to be converted to a more Mac friendly format first. Start by converting the image to DMG format. In my case, I'm converting a DBAN ISO, but substitute these values with the file you want to convert.

$ hdiutil convert ~/dban-2.3.0_i586.iso -format UDRW -o ~/dban-2.3.0_i586.dmg
Reading DBAN                             (Apple_ISO : 0)…
.............................................................................
Elapsed Time:  1.172s
Speed: 13.6Mbytes/sec
Savings: 0.0%
created: /Users/myaccount/src/os/dban-2.3.0_i586.dmg

Next, plug in the USB drive and list all the available disks.

$ diskutil list
/dev/disk0
   #:                       TYPE NAME                  SIZE       IDENTIFIER
   0:      GUID_partition_scheme                      *250.1 GB   disk0
   1:                        EFI EFI                   209.7 MB   disk0s1
   2:                  Apple_HFS OS X                  249.7 GB   disk0s2
/dev/disk1
   #:                       TYPE NAME                  SIZE       IDENTIFIER
   0:     Apple_partition_scheme                      *17.4 MB    disk1
   1:        Apple_partition_map                       32.3 KB    disk1s1
   2:                  Apple_HFS Flash Player          17.4 MB    disk1s2
/dev/disk2
   #:                       TYPE NAME                  SIZE       IDENTIFIER
   0:     Apple_partition_scheme                      *18.1 MB    disk2
   1:        Apple_partition_map                       32.3 KB    disk2s1
   2:                  Apple_HFS Flash Player          18.1 MB    disk2s2
/dev/disk3
   #:                       TYPE NAME                  SIZE       IDENTIFIER
   0:     FDisk_partition_scheme                      *8.0 GB     disk3
   1:                 DOS_FAT_32 UNTITLED 1            8.0 GB     disk3s1

In my case, my USB drive is /dev/disk3. Now unmount that drive, but leave it plugged into the system.

$ diskutil unmountDisk /dev/disk3
Unmount of all volumes on disk3 was successful

Write the image to disk using the dd command.

$ sudo dd if=dban-2.3.0_i586.dmg of=/dev/disk3 bs=1m
Password:
15+1 records in
15+1 records out
16719872 bytes transferred in 6.551918 secs (2551905 bytes/sec)

Finally, eject the disk.

$ diskutil eject /dev/disk3
Disk /dev/disk3 ejected

Change the Active Windows Firewall Profile

The Windows Firewall uses three different profiles: 1) Domain, 2) Private, and 3) Public. The system will assign a profile based on the network it is connected to. The Domain profile is only used when the computer is a member of a domain and a domain controller can be reached. The Private profile is intended for trusted networks, such as at home or in a work environment where there is not a domain. The Public profile is for use on untrusted networks, such as in public places. Each profile has a slightly different default set of rules and can be customized individually.

The Windows Firewall with Advanced Security on Local Computer properties window shows the status of each firewall profile and which profile is currently active.

Unfortunately, there is no easy way to use the GUI to change the active profile. To do that, we'll use Powershell. Too see the active firewall profile, run:

> Get-NetConnectionProfile

Name             : Network  2
InterfaceAlias   : Integrated
InterfaceIndex   : 8
NetworkCategory  : Public
IPv4Connectivity : Internet
IPv6Connectivity : NoTraffic

To change the network profile, you'll have to run Powershell as administrator.

> Get-NetConnectionProfile | Set-NetConnectionProfile -NetworkCategory Private

The Windows Firewall with Advanced Security on Local Computer properties window now shows the Private profile as active.

Redirect Command Line Output to Files in Linux

When using a Linux shell, often you'll find it useful to write the output of a command to a file instead of to the screen. This could be to edit a configuration file or simply to save the output for later use. There are a couple of ways to do this and each has its use.

To redirect output to a file, you can use the carrot (>). Keep in mind, this will overwrite the file if it already exists or create the file if it does not already exist.

$ echo "Hello world!" > ~/output.txt
$ cat ~/output.txt 
Hello world!

If you use the carrot (>) again to write something to the same file, the original contents will be overwritten (I know I said this twice, that was deliberate). If you want to keep the contents of an existing file, you can use the double carrot (>>) to append the output at the end of the file. The double carrot (>>) will also create the file if it does not already exist.

$ echo "Hello again world!" >> ~/output.txt
$ cat ~/output.txt 
Hello world!
Helo again world!

If you want to write the output of a command to both the screen and to a file, use the tee command. By default tee will overwrite an existing file if it exists and create the file if it does not exist.

$ echo "Hello world!" | tee ~/output.txt
Hello world!
$ cat ~/output.txt 
Hello world!

When using tee, if you want to append to the end of an existing file instead of overwriting it, use the -a flag.

$ echo "Hello again world!" | tee -a ~/output.txt
Hello again world!
$ cat ~/output.txt 
Hello world!
Helo again world!

Remove Services from Windows

Ever had a Windows services that either wasn't working correctly or for some reason didn't get removed correctly when uninstalling or changing an installed application? For cases like these, the sc command can help to get things cleaned up. sc is a command line utility for interfacing with the Service Control Manager. To remove the service, first you'll have to know its name. Launch services.msc and find the service you want to remove. Open up the properties dialog for it and note what is displayed next to "Service name." This is the value you'll need to put into the command line.

Run cmd.exe as administrator. This is the syntax for removing a service:

>sc delete <service name>

Downloading Files Using the Command Line

If you're used to using a GUI, you probably never think about how to download content from the Internet or a local server. Downloading content using the command line, however, takes a little more conscious effort. Luckily, its pretty easy to do once you get the hang of it. For me, there are two commands that I like to use most of the time and they are wget and curl.

wget
wget is a petty straight forward in how it works: it downloads things. It can be used to download files using HTTP, HTTPs, and FTP. I find wget useful for either downloading files to a server that might not have a GUI or if I want to script something. As an example, to download the latest version (as of this writing) of Firefox

$ wget https://ftp.mozilla.org/pub/firefox/releases/44.0.2/linux-x86_64/en-US/firefox-44.0.2.tar.bz2

curl
curl downloads and tries to process the target all in one action. curl also supports a lot more protocols than wget. This tool is great if you want read a file stored on a server (among other things). My favorite use for it is actually pretty simple: seeing my external IP address

$ curl icanhazip.com

Enable SSH Key Based Authentication

By default when you setup an SSH server, users will login using username/password authentication, just like if they were physically present at the box. SSH also supports using keys to authenticate. This can by useful for a number of reasons, such as:

• Make logging in faster
• Scripting and automation
• Part of mult-factor authentication

Before we go any further, log into your SSH server to make sure it is configured to allows key based authentication.

$ ssh myaccount@server
$ cat /etc/ssh/sshd_config | grep AuthenticationMethods

The default configuration for SSH servers is for that command to return nothing, which means key based authentication is enabled. If it returns publickey, it will also work. There may be other values included in the line. If they are separated by a comma, then they require multi-factor authentication. If they are separated by a space, they can be used for single-factor authentication.

In order to make things work, you need to generate a key-pair for your user account on the endpoint (client) you are going to connect from. Then, you need to put the public key from that key-pair onto the system where you are logging in using SSH (server). If you want to have multiple different users be able to log in using keys, you need to repeat this process for each of them. Let's walk through it.

Start by generating the key-pair using your account on the client system. You can adjust the options here depending on how strong of a key you want to use. I'm going to use the strongest possible key that is supported at this time.

$ ssh-keygen -b 521 -E sha256 -o -t ecdsa

Let's have a look at where the new keys are stored

$ ls -l ~/.ssh/
total 16
-rw------- 1 myaccount myaccount  736 Feb  6 15:33 id_ecdsa
-rw-r--r-- 1 myaccount myaccount  268 Feb  6 15:33 id_ecdsa.pub
-rw------- 1 myaccount myaccount 1386 Feb  8 20:46 known_hosts

In this case, id_ecdsa is my private key. Notice how only myaccount has access to them. The public key is id_ecdsa.pub and anyone can read it. Next, you'll have to get a copy of the public key over to the SSH server and store it in the home directory for the user account you'll be logging into. We can use scp to get the file copied over.

$ scp ~/.ssh/id_ecdsa.pub myaccount@server:~/

This will place the public key in the home directory of your account on the SSH server. For the next part, you'll have to connect to the SSH server.

$ ssh myaccount@server

Copy the public key to the end of the authorized_keys file for the account. Note the use of the double carrot (>>) here. A single carrot (>) will overwrite the file if it already exists. A double (>>) carrot will add text to the end of the file if it already exists. Either way will create the file if it does not already exist.

$ cat id_ecdsa.pub >> ~/.ssh/authorized_keys

Now you're all set. Log out of your SSH session and then log back in. When you reconnect, it should authenticate with your key-pair and not prompt for a password (unless the server is configured to use multi-factor authentication).

Solving Over The Wire Bandit Levels 0 - 5

The Over The Wire Bandit challenges are a cool way to learn your way around a Linux shell. Some of the levels are easier to figure out than others. I highly encourage working through these challenges on your own. If you get really stuck to the point of giving up, then read on. Also, some of the challenges can be solved multiple ways. This is how I solved them.

Level 0
This level is about getting logged into the system using ssh. Log in with the username bandit0 and password bandit0 as provided on the page

$ ssh -l bandit0 bandit.labs.overthewire.org

Level 0 -> 1
This level is about learning to read files. The password is stored in a file called readme stored in the home directory

$ cat readme 
boJ9jbbUNNfktd78OOpsqOltutMc3MY1

Level 1 -> 2
For the next and all subsequent levels, you'll probably want to exit out of the ssh connection and then reconnect using the next sequentially higher username (bandit1, bandit2, bandit3, etc.) along with the password unlocked in the previous challenge.

This challenge is figuring out how to read a file that starts with special character, in this case a hyphen (-). This can be solved by specifying the full path to the file

$ cat ./-
CV1DtqXWVFXTvM2F0k09SHz0YwRINYA9

Level 2 -> 3
This level is learning how to handle spaces in a file name. The answer is by using the backslash (\) character. Also, once you start typing the name, you can use the tab key to autocomplete the name of the file and the backslashes are put in automatically

$ cat spaces\ in\ this\ filename 
UmHadQclWmgdLOKQ3YNgjWxGoRMb5luK

Level 3 -> 4
This level is learning how to look around the directory structure and find hidden files. The password is stored in the inhere directory. Start by having a look around

$ ls
inhere
$ ls -a inhere/
.  ..  .hidden
$ cat inhere/.hidden 
pIwrPrtPN36QITSp3EQaw936yaFoFgAB

Level 4 -> 5
All of the files in the inhere directory are in binary format (not human unreadable), except one. Sure, you can just read all of them until you find the correct file, but there's a better way to do it using the file command to see what type of files they are

$ file inhere/*
./-file00: data
./-file01: data
./-file02: data
./-file03: data
./-file04: data
./-file05: data
./-file06: data
./-file07: ASCII text
./-file08: data
./-file09: data
$ cat inhere/-file07 
koReBOKuIDDepwhWk7jZC0RTdopnAYKh

Generate an ECDSA P521 SSH Host Key

Elliptic curve cryptography provides stronger protection with smaller keys when compared to non-elliptic curve algorithms. Let's start by seeing what key-pairs already exist

$ ls /etc/ssh | grep .pub
ssh_host_dsa_key
ssh_host_dsa_key.pub
ssh_host_ecdsa_key
ssh_host_ecdsa_key.pub
ssh_host_ed25519_key
ssh_host_ed25519_key.pub
ssh_host_rsa_key
ssh_host_rsa_key.pub

There is already one elliptic curve key-pair. Let's see how many bits it is using

$ cat ssh_host_ecdsa_key.pub | cut -d" " -f1
ecdsa-sha2-nistp256

This means the key-pair uses the ECDSA algorithm with a SHA2 hash and the NIST P256 curve. We can do better than that. Start by generating a new key-pair using the P521 curve

# ssh-keygen -b 521 -o -t ecdsa -f /etc/ssh/ssh_host_ecdsa_p521_key

The -b option specifies the number of bits to use for the key and 521 is the highest OpenSSH supports right now. The -o option saves the keys in a newer format that is more resistant to brute-force password attempts, but is not supported on versions of OpenSSH prior to 6.5. The -t option specifies the type of key to create. The -f option is the name and location for the new key-pair. The name specified is for the private key. The corresponding public key will also be generated in the same directory.

Now you are going to have to edit your SSH daemon configuration file in order to use the new key. Before making any changes, backup the original configuration

# cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak

Now let's take a look at the existing host keys that are configured

$ cat sshd_config | grep HostKey
# HostKeys for protocol version 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key

You'll have to add an entry for the newly created key. If you like to keep your config files organized, you can use your text editor of choice to add the new entry with the others. This will add a new line at the bottom of the file for the new key

# echo HostKey /etc/ssh/ssh_host_ecdsa_p521_key >> /etc/ssh/sshd_config

Restart the SSH daemon to make the changes take effect

# /etc/init.d/ssh restart

Now your OpenSSH server is ready to use the newly created key-pair.

Grant Sudo Permissions in Linux

To grant sudo powers to a user account, use the usermod command

# usermod -a -G sudo <username>

The -a flag specifies that the user should be added to a group. The -G flag is used to specify which group the user should be added to, in this case it is the sudo group.

Alternatively, the adduser command can be used to accomplish the same objective

# adduser <username> sudo

This will add the specified user to the specified group.

To remove a user from the sudo group, run the deluser command

# deluser <username> sudo

It should be noted that only root has permission to manage the sudo group, so these commands will have to be performed either by a user who already has sudo powers, or by the root user.

Set a Static IP Address in Debian Based Linux

Out of the box, most systems seem to be configured to use DHCP to automatically obtain an IP address. For laptops/desktops, this is usually fine. For servers or any system that is going to accept incoming connections, setting a static IP address will make things easier. These instructions should apply to any Debian based Linux distribution, to include Ubuntu, Mint, Kali, etc.

The networking configuration is located at

/etc/network/interfaces

Let's look at the default contents of the file

$ cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
allow-hotplug eth0
iface eth0 inet dhcp

Before changing the default configuration, make a copy just in case something gets messed up

$ sudo cp /etc/network/interfaces /etc/network/interfaces.bak

The part of the file that matter is the last line. I like to leave the old content there, commented out. The last line of the file looked like this

iface eth0 inet dhcp

And now I've updated that line and added some additional configuration so now it looks like this

#iface eth0 inet dhcp
iface eth0 inet static
       address 192.168.1.5
       gateway 192.168.1.1
       netmask 255.255.255.0
       network 192.168.1.0
       broadcast 192.168.1.255

You'll have to update those values based on your network setup. Now restart the networking daemon so the new settings take effect

$ sudo /etc/init.d/networking restart

Verify the new configuration using ifconfig

$ /sbin/ifconfig eth0