NGINX Basic Authorization

 

 I wanted to quickly restrict access to a web domain running on NGINX. An easy way to do this is:

1. find your configuration file often in a subdirectory of /etc/nginx/ for me the path was /etc/nginx/conf.d/local.conf

2. create a file called .htpasswd in the same (could be elsewhere) directory and follow the instructions here as to what to put in it - don't bother with the .htaccess file.

3. add the following lines to your NGINX configuration file and read about them here - you only add a location to pinpoint restricted areas (without means blanket authorization required):

    auth_basic           "closed site";
    auth_basic_user_file conf/htpasswd; 

Hot Chicken Salad

Marinate chicken breast in:

soy sauce (sparingly used!)

Sambal Olek

Teaspoon of honey

Garlic

Prepare Salad

Roughly chop:

Basil

Roquet

Mint

Finely chop:

Chilli

Lemon Grass

Crushed clove garlic

Quartered (or divided further if big) tomato

Squeeze half a lime over the top

Cook chicken in hot oil on griddle. Add to Salad.

libGL error: failed to load driver: swrast

This error occurred for me running Ubuntu 14.04 LTS with nvidia-331-updates proprietary driver.

SOLUTION

when you run the command  

ldconfig -p | grep libGL.so

you want the output to look like this:

    libGL.so.1 (libc6,x86-64) => /usr/lib/nvidia-331-updates/libGL.so.1

    libGL.so.1 (libc6) => /usr/lib32/nvidia-331-updates/libGL.so.1
    libGL.so (libc6,x86-64) => /usr/lib/nvidia-331-updates/libGL.so
    libGL.so (libc6) => /usr/lib32/nvidia-331-updates/libGL.so

but it doesn't.

go to

cd /etc/ld.so.conf.d

list all of your files in there

ls

check each file

nano filename.conf 

if the file has a line pointing to "/usr/lib/i386-linux-gnu/mesa" replace with "/usr/lib32" or delete the file if another file in the directory already has this then:

sudo ldconfig 

and check the output of

ldconfig -p | grep libGL.so 

attribution: http://askubuntu.com/questions/313173/why-are-my-32bit-opengl-libraries-pointing-to-mesa-instead-of-nvidia-and-how-do

Web scraping databases with Ruby

Web scraping databases with Ruby

Introduction

This post is more of an overview with issues that I have encountered and my way of approaching accumulating data records from websites than a technical how-to. For technical information read the documentation and consult stackoverflow or other forums. I do provide a neat way to turn your json records into csv courtesy of an unattributed forum post (if I find it again, I will attribute).

Ingredients

1. Ruby 2.1 or up;
2. mechanize for Ruby;
3. nokogiri;
4. json gem. 

Alternatively to point 2. above you will need the following bundle:

1. watir
2. watir-webdriver
3. headless

Method

1. Examine your target database - you can usually search by some identifier number. What range of numbers does the database employ? Does it count up from 1 or 1000? Are there series of numbers associated with a type of record as part of the set you are acquiring (e.g. individual or corporation)? Does the range have zeros prepended?
2. Determine the range of numbers you wish to search.
3. Divide this up into manageable sub-ranges if the numbers exceed say 10,000 queries. Or if there are 20 million queries, perhaps you do 20 sub-ranges of a million each.
4. Test getting to a sample target record in irb with mechanize first. Can you successfully submit the form? Save the result and examine what appears in your browser.
5. If you can get to the target record with mechanize then you have two options: save the html for each record and post-process; or process on the fly. Either way you will be using nokogiri and xpath selectors.
6. If you can't figure it out with mechanize (usually because there is a whole pile of javascript going on), then you will have to use the webdriver headless option: start off without headless and test with irb. You may well have to process on the fly, so monitor your records as they come through and restart if something is amiss.
7. Process your data with nokogiri and xpath selectors. Using Firebug will help you visualise where things are. Remember: nokogiri structures everything into an onion of associative arrays (hashes).
8. Changing the xml tags with nokogiri can sometimes be the silver bullet to help you solve a problem of logic in extracting data!
9. Conditional statements are your friend. There may be 5 potential rows of data but only four, three or two appear in various records in your range. Can you use conditional statements about label text or distinguishing tags to help you?
10. Save your records as one record per file by the number you searched for in json format as a hash mimicking the website record. You can convert this to csv later for presentation in spreadsheets with the following:

require 'json'
require 'csv'
ary = Array.new

Dir.foreach("data") do |f|
  next if f == '.' or f == '..'
  # put the file into the hash
  record_json = File.open("data#{f}", "r").read
  record = JSON.parse(record_json)

  ary << record 
end

CSV.open("data/data.csv", "wb") do |csv|
  csv << ary.first.keys
  ary.each do |hash|
    csv << hash.values
  end
end

Chilli Chicken Variation # 3 #paleo

Base:
Butter, onion, sambal oeleck, mustard powder, ground coriander.
"Simmer" on low heat as if to caramelise. Add boiling water as required to prevent burning.

Chicken:
Cut strips of thigh and coat in coconut flour.

Cook:
Braise chicken in base - still on low heat. Add very small amounts of boiling water if required. Add small splashes of maple syrup after chicken is mostly cooked. Then continue to braise for a bit longer than you would expect due to low heat. Chicken juices should eventually moisten the base.

Serve:
Boiled baby carrots and baby corn between roquet garnish around edge of plate.

Fruit tipple

Fresh Orange Juice
Kiwi Fruit
Rock Melon

In the blender.

Add white rum and ice, makes its own daiquiri!

Summer has arrived!

And to celebrate I'm posting an old classic:
Prosciutto
Rock Melon
Tooth picks

Wrap the prosciutto around the rock melon and put a tooth pick in it. How easy is that!?

Another gluten-free dairy-free biscuit - coconut flavoured

1 cup desiccated coconut
1 cup gluten-free all purpose plain flour
3/4 cup caster sugar
1/4 cup icing sugar
Pinch of gluten-free baking powder
60g melted copha
2 beaten eggs
Generous splash of coconut essence
Hint of vanilla essence

Amounts above may be innaccurate.

Bake at 170 C fanforced for 20 mins

Variation allegretto on Stephanie Alexander's slow-roasted loin of pork with garlic, rosemary and fennel

We had fennel, celery and some starting-to-grow-old granny smith apples from my recently commenced participation in an organic fruit and vegetable bulk buy and split it up type arrangement.

I followed the recipe in Stephenie Alexander's the cook's companion except as follows:

1. Cut of Pork was pork rib roast (no ribs) aka pork roast aka pork centre loin roast -I ended up cutting it open a bit more to smear the oil mixture over a greater area;
2. Cooked at 250 celsius for 15 minutes, then turned heat down to 200 for further 45 minutes;
3. Absolutely molested the skin with salt to create crackling NOTE: remove crust of salt upon beong cooked;
4. No pork rib bones in the  vegetables;
5. Substitue potatoes with granny smith apples.

Flat 2 late breakfast

Semi poached eggs (2 cms of water in pan and dash of vinegar)
Grilled bacon
Dark rye toast
Avocado butter: parsley, tomato relish whizzed avocado

Gluten-Free Dairy-Free Strawberry Friands

These friands are gluten and dairy free and taste like cumulus at dawn. It's strawberry season so I used fresh strawberries but you could put any kind of berry in - even something frozen like a frozen raspberry. This recipe makes about twelve in small shallow muffin baking thingo.

You will need:
1.5 cups pure icing sugar
1.5 cups almond meal
0.5 cup gluten free plain flour
3 eggs
180 g Copha
Fresh strawberries

Melt the Copha on the stove and prepare some ice water in a high rimmed baking tray to stand the pan in after the Copha has melted.
While Copha is cooling brush your baking holes with some of the Copha.
Mix sifted icing sugar, almond meal and gluten free plain flower. I cheat and don't sift and use a stick mixer to combine.
Create a well in the dry ingredients and add whisked eggs and cooled Copha.
Mix with a wooden spoon to being with and then bring in the stick mixer.
Fill baking holes to the brim with mixture.
Cut strawberries so as to fit in the centre of the holes. And put in slices of strawberries.
Bake for twenty minutes plus at 180 degrees celsius fan forced.

Allow to stand and dust tops with icng sugar.

Salmone al udon e insalata di alghe

Teriyaki Salmon
Udon Noodles
Salad:
Grated carrot
Grated beetroot
Cob lettuce leaves
Alfalfa sprouts
Continental parsley
Yakinori seaweed shredded

Kevin Rudd Brunch Plate

I thought his breakfast looked comically like Kevin. The only thing missing is his tatty hair cut. What a buffoon!  Anyway the mouth is sirloin pork. The cheeks, eyes and glasses are toast, eggs, salami, amd cherry tomato. The nose is fresh basil (perennial) leaves.

Winter's day beach house salad

Freshly baked salmon.
Shallots.
Cherry tomatoes
Continental parsley
Roquet
Spinach
Olive oil and houmous dressing

Best cold dinner

Brown bread toast and pate. Camembert. Prosciutto.

Burger

The Meat:
Onion Soup Mix; wet bread; salt and pepper; medium grade beef mince so that there is some fat in it; egg.

Caramelised onions.

Melt the cheese on the last turn of the meat.

Lettuce.

Tomato.

Sliced dill pickle.

Hot English mustard and tomato sauce.

Toast the buns in the oven.

Caramelising Onion

More butter than you think you need
Lowest heat burner
Thinly sliced onions
Lots of time

Poached Eggs and Avocado Mash

Poached Eggs
Avocado Mash: mash avocados with a fork; chopped mini roma tomatoes; crushed clove of garlic; squeeze of fresh lime; salt; pepper
Brown bread toast

Brunch Plate de Burgh

Bacon
Egg
Chilli-con-carne: half a bulb of garlic, 6 small red chillies, fried in olive oil, add spices (smokey paprika, sweet paprika, cumin seeds, ground cumin, ground coriander seeds), fry beef mince, add tomato paste, add red kidney beans, add crushed tomatoes, plus water and simmer for ages.
Salsa: green capsicum, red onion, mini roma tomatoes, lime juice
Croissant

Cold lamb lunch

Lamb plus what you see in the photo.