Awk

Understanding AWK: A Beginner’s Guide with Examples

AWK is a powerful programming language designed for text processing and typically used as a data extraction and reporting tool. Named after its creators (Aho, Weinberger, and Kernighan), AWK is a versatile tool that can handle complex text-processing tasks with ease. In this post, we’ll explore three AWK commands, ranging from easy to advanced, using specific file contents and showing the output of each command. Let’s dive in!

Example 1: Easy - Extracting Specific Columns

Target File: ‘data.txt’

Lets say the above file contains the following lines:
Name   Age   Occupation
Alice     30   Engineer
Bob      25   Designer
Carol    27    Teacher

Awk Command

awk '{print $1, $3}' data.txt

Output

Name Occupation
Alice Engineer
Bob Designer
Carol Teacher

Explanation

• awk : The command invokes the AWK programming language.
• ’{ }’ : The curly braces contain the AWK command to be executed.
• print $1, $3 : print is and AWK function that prints text, $1 and $3 represent the first and third columns of the input file, respectively. Each $ followed by a number refers to a specific column in the input data.
• data.txt : The name of the input file to be processed.

This simple command uses AWK to print the first and third columns of data.txt. The {print $1, $3} part of the command tells AWK to display the first and third fields (columns) for each line of the file. This is a great way to quickly extract specific information from a structured text file.

Example 2: Medium - Summing Values in a Column

Target File: ’expenses.txt'

The above file contains the following lines:
Item        Cost
Groceries    100
Utilities    150
Rent      1200
Transport    80
Entertain    200

Awk Command

awk 'NR > 1 {sum += $2} END {print "Total Expense:", sum}' expenses.txt

Output

Total Expense: 1730

Explanation

• awk : Invokes the AWK programming language
• NR > 1 : NR is a built-in AWK variable that represents the current record (line) number. This condition ensures that the action is only performed for the lines where NR is greater than 1, effectively skipping the header row.
• {sum += $2} : Within the curly braces, sum += $2 adds the value in the second column ($2) to the variable sum for each line. kmm* END {print “Total Expense:”, sum} : The END block is execute after all input lines have been processed, it prints the total sum of the values in the second column.
• expenses.txt : The name of the input file to be processed.

This command sums up the values in the second column of expenses.txt. The NR > 1 condition skips the header row. The sum += $2 part adds each value in the second column to the sum variable. Finally, END {print “Total Expense:”, sum} prints the total expense after processing all lines.

Example 3: Hard/Cool - Finding the longest Line in a file

Target File: ‘paragraphs.txt’

The above file contains the following paragraphs:
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum consectetur nunc sit amet risus varius, vel facilisis velit tincidunt. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. In quis luctus libero.

Awk Command

awk '{ if (length($0) > max) { max = length($0); longest = $0 } } END { print "Longest line:", longest }' paragraphs.txt

Output

Longest line: Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas.

Explanation

• awk : The command invokes the AWK programming language.
• { if (length($0) > max) { max = length($0); longest = $0 } } : Within the curly braces, this block of code is executed for each input line.
  • length($0) : The length function returns the length of the current line ($0 represents the entire line).
  • if (length($0) > max) : This condition checks if the length of the current line is greater than the current value of max.
  • max = length($0) : If the condition is true, max is updated to the length of the current line.
  • longest = $0 : The lonngest variable is updated to store the current line.
• END { print “Longest line:”, longest } : The END block is executed after all input lines have been processed. It prings the longest line found.
• paragraphs.txt : The name of the input file to be processed.

This advanced command finds the longest line in paragraphs.txt. The length($0) function calculates the length of each line. The if (length($0) > max) condition checks if the current line is longer than the previously recorded maximum length. If it is, the max variable is updated to the current line’s length, and the longest variable stores the current line. After processing all lines, the END { print “Longest line:”, longest } block prints the longest line.

Conclusion:

AWK is a powerful and flexible tool for text processing that can handle a wide range of tasks. From extracting specific columns to summing values and finding the longest line, AWK commands can be simple or complex, depending on your needs. These examples provide a glimpse into what AWK can do, but there’s much more to explore. Dive into AWK, and you’ll find it an invaluable tool in your text-processing toolkit!

The Bash Prompt

Customizing Your Bash Prompt

Is your bash prompt so ugly it makes you sad? Well look no further, in this article I am going to show you how to make your prompt the envy of the neighborhood.

The Prompt

The Bash prompt is the command-line interface feature in Unix-like operating systems, it is provided by the Bash shell (Bourne Again SHell). The prompt is the area where the user types commands and interacts with the system. It typically appears as a string of characters followed by a cursor, signaling that the shell is ready to accept input.

The PS variable (1-4)

The PS1 variable is what you use to create your primary prompt, this is the prompt most people think of and it is the prompt that lets the user know that the shell is ready to accept commands.

The PS2 variable is the secondary prompt, we won’t spend time on this, just be aware that it is the prompt that appears when the shell expects additional input to complete a command, for example when there is an unmatched parenthesis or quotation mark.

The PS3 variable is used with the select command, often used with a case statement. This is commonly used in the creation of menu scripts.

Lastly, the PS4 variable is used in conjunction with the xtrace, or -x, option and is displayed before each command is executed.

Stock Prompt

When you first install and open the Bash shell, the default prompt can be rather uninspiring, providing minimal information and lacking aesthetic appeal, it will most likely look something like this:

Custom Prompt

Fortunately, customizing the Bash prompt is a simple yet effective way to personalize your command-line experience. By incorporating different character sequences, you can not only enhance the visual appeal but also display relevant information that suits your preferences. In the following steps, I’ll guide you through the basics of customizing your Bash prompt and soon, you’ll find yourself navigating the command line with a tailored and visually pleasing prompt that suits your style and preferences, something similar to this:

The Character Sequences

To make changes to the prompt we need to use character sequences. Character sequences are used to change the appearance of your prompt, including color settings for text and background, and style settings for text including underline, bold, blinking, and bright

Before we begin, there are 2 character sequences you need to be aware of before you attempt to style your prompt and they are as follows:

\[

and

\]

These two sequences are important because they are used to enclose and indicate non-printing sequences. While these styling sequences are used to add color and style or other escape sequences to your prompt, they do not actually take up space on the command line so when you include these escape/style sequences without enclosing them in \[ and \], Bash may miscalculate the length of the prompt leading to misplacement of the cursor and this leads to character printing issues and other weird functional issues. By enclosing the non-printing sequences with \[ \] you tell the terminal to ignore these sequences when calculating the length of the prompt to ensure proper cursor placement and avoid unexpected behavior.

So take care to note in the following examples the escape sequences and the printed components of the prompts and which are wrapped and which are not.

Colors

Color is one customization you can use to create your ideal prompt, the following is a list of the color character codes, they are used in a sequence like this:

\[\e[32m\]

So for example:

PS1="\[\e[31m\]This is red text\[\e[m\] "  

Will create a prompt that looks like this:
This text red text.

While:

PS1="\[\e[41m\]This is red background\[\e[m\] "

Will create a prompt that looks like this:
This is red background

Style

Text color is nice, but what if you want to dress it up a little more, you can add a little style with an underline, bright/bold colors, dim colors, blinking text, or reversed text using the following codes:

Apply the style character your would lke to use as follows:

Underline:

\[\e[4;32m\]

So for example:

PS1="\[\e[4;32m\]This is green underlined text\[\e[m\] "  

will create a prompt that looks like this:
This is green underlined text

Bright:

\[\e[1;32m\]

So for example:

PS1="\[\e[1;32m\]This is bright green text\[\e[m\] "

will create a prompt that looks like this: (depending on your terminal colors and settings)
This is bright green text

note the 4; or the 1; in the character sequenct, the 1; is for bright, the 4; is for underline.

Information

Now that you know how to add color and style to your prompt, lets add the real important stuff, the stuff that gives you the needed information about where you are, what you are, who you are, and more.
The following table is a list of charater sequences and the description of what they display:

So now that we have a nice list of the different characters you can use and what they will display, how do you use them? Well it is simple, say you want your prompt to show the name of the current user in white and the current working directory basename in red, that would look like this:

PS1="\u \[\e[31m\]\W\[\e[m\] "

With this configuration, your prompt will look like this:
(I will use Jake as my user name, and home as the current dir)
Jake ~

Or say you don’t care about color and you just want to display the date, just do this:

PS1="\d : "

This will display a prompt that looks like this:
Wednesday May 13 :

Now that we have a good idea of what character sequences do and how to use them the question becomes, where do we configure this? There are a couple of ways you can implement your own prompt. First, if you are just looking to change the prompt for the current session, you can just open your terminal, and enter the following:

bash-5.25: PS1="<enter your prompt configuration here>"

This will change your prompt for the current session, but it will return to normal once you close your terminal. If you want to change it permanently, all you have to do is set the PS1 variable to whatever you like in either your .bash_profile, or in your .bashrc (you can use your .profile as well, if you prefer, but the other 2 files are more commonly used). Once set in one of the bash files, just close your terminal and reopen it or use the following command, and your prompt will be permanently set, at least set until your edti your PS1 variable again.

source%

Now take your new knowledge and understanding of the PS1 variable and customize and configure to your hearts desire.

Installing Void Linux

Void Linux Install with BtrFS and encryption

So you want to install Void Linux, well you’ve come to the right place, so without further ado, let’s get started.

Section 1 - Select and Prepare the Device

!! NOTE: !! -ALL COMMANDS IN THIS TUTORIAL ARE RUN AS THE ROOT USER, IF YOU ARE NOT ROOT, BE SURE TO EITHER USE SUDO OR CHANGE TO ROOT

This section will cover partitioning your drive and adding encryption. There are many tools avalable to partition your drive, tools like cfdisk, parted, gparted, gdisk, etc, we will be using fdisk. As for encryption, we will be using luks1 since as of the writing of this, grub does not fully support luks2.

Step 1 - Partition

From the command line launch fdisk and point it to the device of your choosing, for this example I will use /dev/sdX. once fdisk is launched with the correct device, we need to create 2 parititons, the following are the selections needed:

fdisk /dev/sdX

Create GPT partition table
Command (m for help): g

Create EFI System Partition (ESP)
Command (m for help): n
Partition number (1-128, default 1): (enter for default)
First sector (2048-500118158, default 2048): (enter for default)
Last sector, +/-sectors or +/-size{K,M,G,T,P}...): +200M (choose 128M to 1G)
Do you want to remove the signature? [Y]es/[N]o: y
Command (m for help): t
Partition type or alias (type L to list all): 1

Create the root partition
Command (m for help): n
Partition number (2-128, default 2): (enter for default)
First sector (2099200-500118158, default 2099200): (enter for default)
Last sector, +/-sectors or +/-size{K,M,G,T,P}...): (enter for default)

Write changes to the disk
Command (m for help): w

Step 2 - Encrypt

Now that the device has been partitioned, we are ready to encrypt the root volume.

Again, since grub does not fully support luks2 we will be using luks1 using the following commands.

Encrypt the root volume with the following command:

cryptsetup luksFormat --type luks1 -y /dev/sdX2

It will warn you that this action will overwrite anything on the selected disk, type YES (all caps) to accept, then it will ask you to create and verify a password.

Now the the root volume is encrypted it is locked and cannot be edited, we need to open the root volume and give it a name with the following command: ( I am calling it cryptvoid, you can name it what ever you would like )

cryptsetup open /dev/sdX2 cryptvoid

you will be prompted for the password you just created and once verified, the encrypted root volume will be open.

Step 3 - Format Partitions

Now we are ready to format the paritions, we will make partition 1 the efi partition and install a fat filesystem and parition 2 will be the root partition with BtrFS.

format the efi parititon:

mkfs.fat -F32 -n EFI /dev/sdX1

format the root partition:

mkfs.btrfs -L Void /dev/mapper/cryptvoid

Step 4 - Create BtrFS Subvolumes

Now that we have the disk partitioned and the file systems created, now we need to create the subvolumes on the BtrFS volume.

First we will create a variable to store all the options we want to use, this is not completely necessary but will save on a lot of redundant typing

BTRFS_OPTS="rw,noatime,compress=zstd,discard=async"

• rw = read/write permissions
• noatime = no access time, prevents sys from updating access timestamp every time a file is accessed
• compress = type of compression
  • zstd = using Zstandard compression algorithm
• discard = improve efficiency of solid state drives by allowing them to reclaim unused space
  • async = operation will be performed asyncronously

Now we will mount the top level subvolume.

mount -o $BTRFS_OPTS /dev/mapper/cryptvoid /mnt 

Now that we have the top level subvolume mounted, it is time to create the subvolumes for root, home, and snapshots

Root subvolume

btrfs su cr /mnt/@

Home subvolume

btrfs su cr /mnt/@home

Snapshots subvolume

btrfs su cr /mnt/@snapshots

Unmount cryptvoid

umount /mnt

The last step of prep before we start the base install is to mount all the subvolumes and the EFI partition

Mount the root subvolume

mount -o $BTRFS_OPTS,subvol=@ /dev/mapper/cryptvoid /mnt

We now have our encrypted volume mounted and we need to create mountpoints for home, root, and snapshots

mkdir /mnt/{efi,home,.snapshots}

Mount the home subvolume

mount -o $BTRFS_OPTS,subvol=@home /dev/mapper/cryptvoid /mnt/home

Mount snapshots subvolume

mount -o $BTRFS_OPTS,subvol=@snapshots /dev/mapper/cryptvoid /mnt/.snapshots

While a key function and benefit of BtrFS is snapshots, there are certain directories that don’t need to be included in the snapshots, we will create some nested subvolumes for these specific directories

mkdir -p /mnt/var/cache
btrfs su cr /mnt/var/cache/xbps
btrfs su cr /mnt/var/tmp
btrfs su cr /mnt/srv

Mount the EFI system parition

mount -o rw,noatime /dev/sdX1 /mnt/efi

Check mountpoints and verify they are correct using one of the following 2 commands

df -h

or

lsblk

Section 2 - Installation

With the drive now partitioned, the file system created, and all volumes and subvolumes mounted, we can begin the base installation of Void Linux

Step 1 - Mirror, C library, Architecture, and Base-system metapackage

While this step is not required it is a good idea to choose a mirror close to your location, A list of mirrors can be found at https://xmirror.voidlinux.org/. However; you can just go with the default mirror if you choose to.

The closest mirror to me is Chicago, so in this example, I will use the Chicago mirror, https://mirrors.servercentral.com/voidlinux/.

Aside from which mirror you want to use, there are a few other choices you will need to make, one of these is which C library to use, Void gives the option of using glibc (Gnu C library) or musl (designed to be more lightweight).

• glibc: /current
• musl: /current/musl

There is also the question of architecture, you will need to select which architecture you plan to use from the following list:

• x86_64
• x86_64-musl
• i686
• aarch64
  I will use x86_64 for this example

Once you decide which mirror and library you want to use (we will be using Chicago mirror and glibc library), we can create a couple more variables to make the next steps a little easier, we need to create a variable for our repo, and a variable for our architecture.

REPO=https://mirrors.servercentral.com/voidlinux/current/
ARCH=x86_64

Create directory and copy RSA keys for verifying package integrity

mkdir -p /mnt/var/db/xbps/keys
cp /var/db/xbps/keys/* /mnt/var/db/xbps/keys/

Now we can install the base system along with a few other odds and ends. (this tutorial will install vim and the linux-mainline kernel, if you want stable, install linux instead of linux-mainline)

XBPS_ARCH=$ARCH xbps-install -S -R "$REPO" -r /mnt base-system linux-mainline btrfs-progs cryptsetup vim

Step 2 - Chroot

With the base system installed we are ready to chroot into our void environment but first we need to mount a pseudo-filesystem needed for chroot

for dir in dev proc sys run; do mount --rbind /$dir /mnt/$dir; mount --make-rslave /mnt/$dir; done

Copy dns configuration into new root so xbps can download new packages inside

cp /etc/resolve.conf /mnt/etc/

chroot into system

BTRFS_OPTS=$BTRFS_OPTS PS1='(chroot) # ' chroot /mnt/ /bin/bash

Step 3 - Installation Configuration

Next we will begin to configure our new install, this will involve setting a hostname, timezone, our hosts, creating an fstab, and more.
First lets look at our rc.conf, you can find and go over the options in this file in the void docs at httsp://docs.voidlinux.org/config/rc-files.conf we can make changes in here but it is not necessary

vim /etc/rc.conf

Set timezone, you can list out all available timezones by running the following command

ls /usr/share/zoneinfo

Once you locate the timezone you need, set the timezone in the /etc/localtime file, I will set the timezone to chicago in this example

ln -sf /usr/share/zoneinfo/America/Chicago /etc/localtime

Next we will set our locale by editing /etc/default/libc-locales and uncommenting your locale. For me this would be en_US.UTF-8 and en_US ISO-8859-1

vim /etc/default/libc-locales

Once you uncomment and save the changes, you need to reconfigure

xbps-reconfigure -f glibc-locales

Set hostname in /etc/hostname
Replace <your chosen hostname> with what ever you want your hostname to be.

echo "<your chosen hostname>" > /etc/hostname

Create the /etc/hosts file, replace myhostname with the hostname you created in the last step.

cat <<EOF > /etc/hosts
#
# /etc/hosts: static lookup table for host names
#
127.0.0.1        localhost
::1              localhost
127.0.1.1        myhostname.localdomain myhostname
EOF

Step 4 - User Management

Now we will move into user management
First we need to create the root password

passwd

Now we can create a new user and give that user a password, add them the necessary groups, and give them sudo privilege
Replace <USER> with the name of the user you are adding

useradd <USER>
passwd <USER>
usermod -aG wheel, (any other groups you want to add user to) <USER>

Now we can change the shell for the root user to bash and edit the sudoers file

chsh -s /bin/bash root
EDITOR=vim visudo

Uncomment the line that says # %wheel ALL=(ALL:ALL) ALL
Then you can also add the following line below the line you just uncommented, this is not required but it is an option

<USER> ALL=(ALL:ALL) ALL

Step 5 - REPOS

Sync repositories

xbps-install -S

This step is not required unless you want to be able to access software that does not have free licenses or if you are a gamer or need 32bit packages

xbps-install void-repo-nonfree
xbps-install -S
xbps-install void-repo-multilib
xbps-install -S

step 6 - Create fstab

Now we can create our fstab, there are multiple ways to do this but we are going to use some command line magic and make this as easy as possible

Create variables for different volumes

EFI_UUID=$(blkid -s UUID -o value /dev/sdX1)
ROOT_UUID=$(blkid -s UUID -o value /dev/mapper/cryptvoid)
LUKS_UUID=$(blkid -s UUID -o value /dev/sdX2)

Next we will use these variables and create and populate our fstab

cat <<EOF > /etc/fstab
UUID=$ROOT_UUID / btrfs $BTRFS_OPTS,subvol=@ 0 1
UUID=$ROOT_UUID /home btrfs $BTRFS_OPTS,subvol=@home 0 2
UUID=$ROOT_UUID /.snapshots btrfs $BTRFS_OPTS,subvol=@snapshots 0 2
UUID=$EFI_UUID /efi vfat defaults,noatime 0 2
tmpfs /tmp tmpfs defaults,nosuid,nodev 0 0
EOF

Step 7 - Install and setup bootloader

We are now on the homestretch, a few more commands and it will be time to reboot

Install the grub package for efi

xbps-install grub-x86_64-efi

Enable encryption

echo GRUB_ENABLE_CRYPTODISK=y >> /etc/default/grub

Edit /etc/default/grub file

vim /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="loglevel=4 rd.auto=1 rd.luks.allow-discards"

Install grub

grub-install --target=x86_64-efi --efi-directory=/efi --bootloader-id="Void"

Section 3 - Miscellaneous

Step 1 - Keyfile (optional)

Create a keyfile to avoid typing passphrase 2x on boot

Create a keyfile out of random data

dd bs=515 count=4 if=/dev/urandom of=/boot/keyfile.bin

Add a 2nd key slot to the LUKS encrypted volume with keyfile.bin as the key

cryptsetup -v luksAddKey /dev/sdX2 /boot/keyfile.bin

Secure the keyfile by setting appropriate permissions

chmod 000 /boot/keyfile.bin 

Allow only root access to /boot

chmod -R g-rwx,o-rwx /boot 

Setup crypttab

cat <<EOF >> /etc/crypttab
cryptvoid UUID=$LUKS_UUID /boot/keyfile.bin luks
EOF

Cofigure dracut to include the keyfile and crypttab in the initial RAM disk

echo 'install_items+=" /boot/keyfile.bin /etc/crypttab "' > /etc/dracut.conf.d/10-crypt.conf
ln -s /etc/sv/dhc /etc/runit/runsvdir/default

Step 2 - Install software/programs/tools

Install software or programs you may want to install at this time

xbps-install <list of desired programs>

Step 3 - Link services

Link services that we want to start on boot
If you want to use a wired connection

ln -s /etc/sv/dhcpcd-eth0 /var/service
ln -s /etc/sv/dhcpcd /var/service

If you want to use wifi you can use wpa-supplicant or you can install Network Manager I prefer to use Network Manager so that is what I will use as an example

xbps-install NetworkManager
ln -s /etc/sv/NetworkManager /var/service

Use xbps to verify installed programs are configured correctly

xbps-reconfigure -fa

Step 4 - Exit and Reboot

Exit chroot

exit

Reboot system

reboot