Buddy Lindseyhttps://buddylindsey.com/2018-03-01T10:00:00-06:00Setup SSL with Static S3 Blog and Site2018-03-01T10:00:00-06:002018-03-01T10:00:00-06:00Buddy Lindseytag:buddylindsey.com,2018-03-01:/setup-ssl-static-s3-blog-site.html<p>Running your static site behind a server certificate on s3 is easy'ish, but very confusing if you don't know how to do it. To start you need to connect s3 to cloudfront to certificate manager then to your dns. It is easier than it sounds.</p><p>We really need to start running more and more sites behind SSL. I have always been a fan of increased security even for things that don't need it.</p>
<blockquote>
<p>"I have nothing to hide, but you don't need to know that."</p>
</blockquote>
<p>I live by that philosophy. As such I want to run all of my sites behind SSL whether it is needed or not. Therefore running my static blog, and sites, need to be secured with SSL. With all the fun configurations you can do with AWS these days you can easily setup a free ssl cert, and run your site on CloudFront making it secure and super fast. All you do is host it on an S3 bucket, and do some connecting of services.</p>
<p>I currently run two static sites in the way described below, and my bill looks to be averaging $1.50 per month. I don't get a lot of traffic, yet, but I don't think it will get too expensive for a while. Besides for AWS it is less about making money on you from s3 storage as it is to get you into the ecosystem ;).</p>
<h2 id="contents">Contents<a class="headerlink" href="#contents" title="Permalink to this headline"><i class="fa fa-link"></i></a></h2>
<ol class='toc'>
<li><a href="#bucket-configuration">Create and Configure S3 Buckets</h2>
<li><a href="#server-certificate">Add Server Certificate</a></li>
<li><a href="#cloudfront-setup">CloudFront Setup</a></li>
<li><a href="#final-dns">Final DNS Setup</a></li>
</ol>
<p>There are 4 main, and several minor, segments to setting up your static site behind an SSL certificate. You create the buckets, configure your certificate, setup CloudFront, and finalize your dns settings.</p>
<p>The following is a screenshot'd walk-through of setting everything up. I set up my farms website and took pictures so you could see the process.</p>
<h2 id="bucket-configuration">Create and Configure S3 Buckets<a class="headerlink" href="#bucket-configuration" title="Permalink to this headline"><i class="fa fa-link"></i></a></h2>
<p>First thing you want to do is creat your buckets. If you go into S3 you will see a button similar to the image below.
<img src="/images/015/01_create_bucket.png" width="650" /></p>
<p>When you click on the button you will get a modal. Here you put in the name of the bucket. Generally people put the name of the url as the bucket name, in this case it is <code>tryingoutdoors.farm</code>. Leave it as <code>US East (N. Virginia)</code> they enable stuff by default other regions do not. I have read of instances where parts of this don't work out of the box outside of <code>US East</code>. If you are worried about regional speed and distribution CloudFront should solve that issue.
<img src="/images/015/02_create_bucket_name.png" width="650" /></p>
<p>After you set your bucket name you will need to set the permissions. Set these to public for everyone to read. It does give a warning about everyone being able to read content from this bucket. Normally, you would heed the warning and not do this, but we know, and are aware, we want to make all content public in this bucket.
<img src="/images/015/03_set_public_permissions.png" width="650" /></p>
<p>Finally, in the modal steps you get to review what you want to do. Make sure the bucket name is the name of your url, <code>tryingoutdoors.farm</code> in this example. Also make sure that public permissions is <code>Enabled</code>. Hit the create/save button and you are good to go.
<img src="/images/015/04_review_bucket.png" width="650" /></p>
<p>Now you should have a public s3 bucket with the name of your domain name. Since it has public read permissions it will probably look like below, as a line item.
<img src="/images/015/05_bucket_line_item.png" width="650" /></p>
<p>Go ahead and click into the bucket, and you will see tabs close to the top with <code>Overview</code>, <code>Properties</code>, <code>Permissions</code>, and <code>Managment</code>. Click on the <code>Properties</code> tab.
<img src="/images/015/06_click_on_bucket_properties.png" width="650" /></p>
<p>In the <code>Properties</code> tab look for the square box that says <code>Static website hosting</code> so we can configure it properly.
<img src="/images/015/07_click_on_static_website_hosting.png" width="650" /></p>
<p>First save off the Endpoint you will need it later. For setup we want to select <code>Use this bucket to host a website</code>. For the <code>Index document</code> we want to use <code>index.html</code> since this will be the html page that will load up as the home page. Once that is done click on <code>save</code>.
<img src="/images/015/08_set_static_hosting_and_index_document.png" width="650" /></p>
<p>Now that we are at this point we are 90% done with setting up a bucket. So far it has been easy stuff. At this point we could set the content policy of the bucket and the endpoint would be good to go and we could start hosting our static site.</p>
<p>However, we want to go ahead and forward <code>www</code> to our root site. To do this we want to follow the following above steps.</p>
<ol>
<li>Create bucket with the <code>www</code> version of your site. Ex. <code>www.tryingoutdoors.farm</code></li>
<li>Don't worry about it being public</li>
<li>Open bucket properties</li>
<li>Click on the <code>Static web hosting</code> box</li>
</ol>
<p>The 4 steps above you have already done so now we just change the last thing we did. We want to select <code>Redirect Requests</code> and add in the <code>url</code> to redirect to. the root of your site. In this case <code>tryingoutdoors.farm</code>. So once we setup DNS for <code>www.tryingoutdoors.farm</code> to point to this bucket it will then redirect to <code>tryingoutdoors.farm</code> instead of loading anything up.
<img src="/images/015/09_www_redirect_to_url.png" width="650" /></p>
<p>Now for the last couple of steps for configuring the bucket we want to go ahead and open up the permissions tab for the <code>tryingoutdoors.farm</code> bucket
<img src="/images/015/10_bucket_permission_policy.png" width="650" /></p>
<p>Next we want to set the <code>Bucket Policy</code> so that we allow anyone to be able to read our content.
<img src="/images/015/11_bucket_policy.png" width="650" /></p>
<p>Here is the code from above you can copy and paste. Just remember to change <code><your_domain>.farm</code> to your domain name.</p>
<div class="highlight"><pre><span></span><span class="p">{</span>
<span class="nt">"Version"</span><span class="p">:</span> <span class="s2">"2012-10-17"</span><span class="p">,</span>
<span class="nt">"Statement"</span><span class="p">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="nt">"Sid"</span><span class="p">:</span> <span class="s2">"PublicReadGetObject"</span><span class="p">,</span>
<span class="nt">"Effect"</span><span class="p">:</span> <span class="s2">"Allow"</span><span class="p">,</span>
<span class="nt">"Principal"</span><span class="p">:</span> <span class="s2">"*"</span><span class="p">,</span>
<span class="nt">"Action"</span><span class="p">:</span> <span class="s2">"s3:GetObject"</span><span class="p">,</span>
<span class="nt">"Resource"</span><span class="p">:</span> <span class="s2">"arn:aws:s3:::<your_domain>.farm/*"</span>
<span class="p">}</span>
<span class="p">]</span>
<span class="p">}</span>
</pre></div>
<p>Finally, take that endpoint url you saved off earlier and open it up in the browser after you upload your <code>index.html</code> file to your domains bucket. You should now see your site working.
<img src="/images/015/12_upload_index_and_view.png" width="650" /></p>
<h2 id="server-certificate">Add Server Certificate<a class="headerlink" href="#server-certificate" title="Permalink to this headline"><i class="fa fa-link"></i></a></h2>
<p>Now it is time to get our free SSL Cert that autorenews and autoreloads on expiration, isn't that cool. First, open up Certificate Manager and you will be presented with a few buttons, key being <code>Request a certificate</code>. Clickt that one.
<img src="/images/015/13_create_aws_certificate.png" width="650" /></p>
<p>You will be presented with a new page to set up your urls you want used with a certificate. In this case we will use the root of our site, and <code>*.</code> to get the rest of our subdomains to work.
<img src="/images/015/14_add_domains_to_cert.png" width="650" /></p>
<p>Next we will select that we want to do <code>DNS Validation</code> instead of email validation.
<img src="/images/015/15_set_dns_validation_for_cert.png" width="500" /></p>
<p>Finally, review that we are doing the urls we are expecting and doing DNS Validation.
<img src="/images/015/16_review_cert_info.png" width="650" /></p>
<p>Once that is done we will be presented dns settings to set with our domains. I recommend you use Route53 for your DNS needs. If you do all you need to do is click the <code>Create record in Route 53</code> to set the proper dns settings.
<img src="/images/015/17_dns_settings_to_set_route_53.png" width="650" /></p>
<p>Wait anywhere from 5 to 30 minutes, usually, and the certficate manager should validate the dns settings. Once it is validated it will issue your certificate, and you are ready to start using it with your other AWS services.
<img src="/images/015/18_validation_successful.png" width="650" /></p>
<p>One of the reasons to have DNS validation set is so that when it comes time to renew the certificate it will "just work". If we do email validation it will send an email to us every year and we would have to remember to click the link in the email, and not forget.</p>
<h2 id="cloudfront-setup">CloudFront Setup<a class="headerlink" href="#cloudfront-setup" title="Permalink to this headline"><i class="fa fa-link"></i></a></h2>
<p>Now comes the glue that brings together the s3 bucket and your new certificate for your domain name.</p>
<p>CloudFront runs as <code>distribution</code>'s which give it a set of settings and instructs our CDN how to do deal with requests. So to start we need to create a new distribution so just simply click on <code>Create Distribution</code>.
<img src="/images/015/19_cloudfront_distribution_button.png" width="650" /></p>
<p>You are going to get a confirmation page, go ahead and click on <code>Get Started</code>.
<img src="/images/015/20_info_get_started_button.png" width="650" /></p>
<p>For <code>Origin Domain Name</code> set the URL of the s3 bucket we created earlier. The rest is fine.
<img src="/images/015/21_set_origin_domain_name_to_s3_bucket.png" width="650" /></p>
<p>Since we want to have <code>http</code> and <code>https</code> requests return back content we need to set our distribution to redirect all <code>http</code> requests to <code>https</code> so something is returned when people don't use <code>http</code>.</p>
<p>Now that we have our request routing figured out we need to permit specific HTTP Verbs. In our case we just want <code>GET</code> and <code>HEAD</code> since we aren't going to be pushing anything to our webserver.
<img src="/images/015/22_redirect_http_to_https.png" width="650" /></p>
<p>Probably the most important part of this section is this, set the <code>Alternate Domain Names (CNAMES)</code> to your domain name. In this case <code>tryingoutdoors.farm</code>. This is going to let our distribution know to use these urls. Also be sure to select <code>Custom SSL Certificate</code>, and choose your cert from the dropdown available.
<img src="/images/015/23_set_alternate_cname_and_use_ssl_cert.png" width="650" /></p>
<p>Finally, find the section <code>Default Root Object</code>, and set it to <code>index.html</code>. This sets our system be like a standard server to load up <code>index.html</code> if <code>/</code> is requested.
<img src="/images/015/24_set_root_object.png" width="650" /></p>
<p>At this point go take a break, get some tea, go for a walk, maybe go to bed for the night. Hopefully, now that you are back in the morning the distribution is up and running. Basically, the distribution can take a while to set up.</p>
<h2 id="final-dns">Final DNS Setup<a class="headerlink" href="#final-dns" title="Permalink to this headline"><i class="fa fa-link"></i></a></h2>
<p>Finally, we are ready to wrap up by pointing our dns for <code>tryingoutdoors.farm</code> to our distribution and <code>www.tryingoutdoors.farm</code> to <code>tryingoutdoors.farm</code>. As noted we are pointing our root url to our CloudFront Distribution that we just setup. We are goint point our <code>www</code> to an s3 bucket so which is configured to redirec to our root site.</p>
<p>First, step is open up <code>Route 53</code> and select your domain.
<img src="/images/015/25_route_53_domain.png" width="300" /></p>
<p>Next we want to click on <code>Create Record Set</code> so we can add our configurations.
<img src="/images/015/26_create_record_set.png" width="650" /></p>
<p>Now on the right side of the page you should see a new window that looks similar to below. Leave the textbox at the top blank so we are setting the route.</p>
<p>Choose type of record as <code>A - IPv4 address</code> and select <code>Yes</code> for Alias. At this point you should be able to click on <code>Alias Target</code> and a dropdown should appear. Just scroll through it and look for your CloudFront Distribution for your url and select it. Then go down and hit save. You have successfully configured the root of the site.
<img src="/images/015/27_set_alias_for_distribution.png" width="650" /></p>
<p>Now got through the same steps to create a new record and point your <code>www</code> to an <code>A - IPv4 address</code> alias of your <code>www</code> s3 bucket in the dropdown and click save.
<img src="/images/015/28_set_www_to_s3.png" width="650" /></p>
<p>Now all you need to do is wait, hopefully not long, as all the DNS updates everywhere. Once your dns is updated try browsing your site and it should be behind an SSL certificate. It should also be blazing fast.</p>
<h2>Final Thoughts</h2>
<p>There were a lot of steps to get this all setup. However, once you get through it once, it is a lot easier, and it makes a few AWS things make a bit more sense. The great thing is you now have an SSL backed site for super cheap compared to paying for a server and a site certificate.</p>
<p>I will say there is one caveat to doing all this to be aware of. When you push a new blog post or update to your site, it might take a bit to load as it propagates out to the world accross the CDNs. It isn't instant like it would be if you were reading directly from an s3 bucket.</p>
<p>With that I hope more people start using SSL on their sites.</p>Replication with PostgreSQL on Ubuntu2018-02-13T10:00:00-06:002018-02-13T10:00:00-06:00Buddy Lindseytag:buddylindsey.com,2018-02-13:/replication-with-postgresql-on-ubuntu.html<p>Learn what it takes to setup a replica server for PostgreSQL on Ubuntu with this step-by-steb guide. If you have never done it before this will help you get over that first hump.</p><p>In this guide, you will learn how to setup replication between two PostgreSQL servers on Ubuntu. Along with the how and why of what you will learn.</p>
<h2 id="contents">Contents<a class="headerlink" href="#contents" title="Permalink to this headline"><i class="fa fa-link"></i></a></h2>
<ol class='toc'>
<li><a href="#why-replication">Why Replication?</h2>
<li><a href="#compatibility">Ubuntu and PostgreSQL Version Compatibility</a></li>
<li><a href="#prerequisites">Install Prerequisites</a></li>
<li><a href="#install">Install PostgreSQL</a></li>
<li><a href="#create-user">Create Replication User</a></li>
<li><a href="#connection-permission">Replica Connection Permission</a></li>
<li><a href="#configure-archive">Configuration for Creating Archive Files for Replication</a></li>
<li><a href="#make-archive-folder">Make Folder for Saving Archive To</a></li>
<li><a href="#restart-primary">Restart Primary Server</a></li>
<li><a href="#replication-server">Replication Server</a></li>
<li><a href="#configure-replica">Configure Replica postgresql.config</a></li>
<li><a href="#sync-data">Sync Primary Data to Replica</a></li>
<li><a href="#recovery-file">Configure recovery.conf for Continuing Replication</a></li>
<li><a href="#restart-replica">Restart Replica Server</a></li>
<li><a href="#test-replication">Test Replciation is Working</a></li>
<li><a href="#final-thoughts">Final Thoughts</a></li>
</ol>
<h2 id="why-replication">Why Replication?<a class="headerlink" href="#why-replication" title="Permalink to this headline"><i class="fa fa-link"></i></a></h2>
<p>Database replication is fairly important when you want to scale your database horizontally. Services like Amazon RDS make it super easy. However, that can be expensive, or as I found in my case, I couldn't use RDS. We ended up needing to use a plugin that wasn't available for RDS so we have to host our own.</p>
<p>The process of setting up replication contains multiple steps, and if you get them wrong it wont work, and wont easily tell you why. Through this guide I will show you what you need to do to setup replication with ubuntu. Also, hopefully, enough information to figure it out for other distributions as well. I also hope to explain what is going on so you can understand the why of what you are doing.</p>
<h2 id="compatibility">Ubuntu and PostgreSQL Version Compatibility<a class="headerlink" href="#compatibility" title="Permalink to this headline"><i class="fa fa-link"></i></a></h2>
<p>Below are the versions of postgres and ubuntu I have tested this guide on to make sure that it works. As new versions of both come out I hope to test them to keep this guide up-to-date.</p>
<table>
<thead>
<tr>
<th></th>
<th align="center"> Ubuntu 14.04 </th>
<th align="center"> Ubuntu 16.04 </th>
<th align="center"> Ubuntu 18.04 </th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>PostgreSQL - 10.0</strong></td>
<td align="center"><i class="fa fa-check-circle fa-green-check"></i></td>
<td align="center"><i class="fa fa-check-circle fa-green-check"></i></td>
<td align="center"><i class="fa fa-question-circle fa-green-check"></i></td>
</tr>
<tr>
<td><strong>PostgreSQL - 9.6</strong></td>
<td align="center"><i class="fa fa-check-circle fa-green-check"></i></td>
<td align="center"><i class="fa fa-check-circle fa-green-check"></i></td>
<td align="center"><i class="fa fa-question-circle fa-green-check"></i></td>
</tr>
<tr>
<td><strong>PostgreSQL - 9.5</strong></td>
<td align="center"><i class="fa fa-check-circle fa-green-check"></i></td>
<td align="center"><i class="fa fa-check-circle fa-green-check"></i></td>
<td align="center"><i class="fa fa-question-circle fa-green-check"></i></td>
</tr>
</tbody>
</table>
<h2 id="prerequisites">Install Prerequisites<a class="headerlink" href="#prerequisites" title="Permalink to this headline"><i class="fa fa-link"></i></a></h2>
<p>With ubuntu we need to add the apt repository so that we can install postgres. One of the frustrating sides to notice in this is the <code>pgdg</code> if you don't get that it wont work. However, you should just be able to copy and paste the following 4 lines and it should work no matter your version of ubuntu.</p>
<div class="highlight"><pre><span></span>sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
sudo apt-get install wget ca-certificates
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
sudo apt-get update
</pre></div>
<h2 id="install">Install PostgreSQL<a class="headerlink" href="#install" title="Permalink to this headline"><i class="fa fa-link"></i></a></h2>
<p>Next is to actually install PostgreSQL. Through the rest of this guide I'll add the line or file you need to use for each version of postgres that you want to use. Only use that line or frustration will happen. Now all we need to do is a simple <code>apt-get install</code> and we are good to go.</p>
<div class="highlight"><pre><span></span>sudo apt-get install postgresql-10
sudo apt-get install postgresql-9.6
sudo apt-get install postgresql-9.5
</pre></div>
<h2 id="create-user">Create Replication User<a class="headerlink" href="#create-user" title="Permalink to this headline"><i class="fa fa-link"></i></a></h2>
<p>Now that Postgres is installed we need to create our user for doing replication. Postgres requires a user with replication privileges in order to pull data over. By convention people use a user called <code>replication</code> with the <code>replication</code> permission.</p>
<p>First thing we need to do is log in to postgres, and the easiest way to do that is to switch users to the <code>postgres</code> user. Once you are the <code>postgres</code> user use <code>psql</code> to log in and execute the <code>CREATE USER</code> command.</p>
<div class="highlight"><pre><span></span>sudo su - postgres
psql
CREATE USER replication REPLICATION LOGIN ENCRYPTED PASSWORD 'password1';
</pre></div>
<p><code>CREATE USER</code> will create the replication user, set the replication permission, set the user to be able to login, and encrypt the password in the database. It is recommended to use a different password than <code>password1</code>. Also please do encrypt the password because if you don't it can lead to headaches later on with other tool configurations.</p>
<h2 id="connection-permission">Replica Connection Permission<a class="headerlink" href="#connection-permission" title="Permalink to this headline"><i class="fa fa-link"></i></a></h2>
<p>Now that we have our user we need to give them permission to actually connect to the primary postgres server. We do that in the <code>pg_hba.conf</code> file. In the file we declare what user, database, and where they are connecting from. In this case our replica server is at ip <code>192.168.1.102</code> so we are explicitly telling our server to allow connections from the user <code>replication</code> from that <code>ip</code> only. The more servers or users you have the more you add to this list.</p>
<div class="files">
/etc/postgresql/10/main/pg_hba.conf<br />
/etc/postgresql/9.6/main/pg_hba.conf<br />
/etc/postgresql/9.5/main/pg_hba.conf
</div>
<div class="highlight"><pre><span></span>host replication replication 192.168.1.102/32 md5
</pre></div>
<h2 id="configure-archive">Configuration for Creating Archive Files for Replication<a class="headerlink" href="#configure-archive" title="Permalink to this headline"><i class="fa fa-link"></i></a></h2>
<p>One of the most critical parts of configuring replicaiton is the saving of data to archive files so they can be pulled over to replicas. The following configuration does just this.</p>
<p>So that you know <code>WAL</code> stand for <code>write-ahead logging</code>. It is very useful for a few things, but we are only going to focus on replication to other servers in this guide.</p>
<p>The first step is to set the <code>wal_level</code>. I liken it to <code>logging_level</code> it is how you configure the intensity, might be a bad analogy, but it is how I keep it in my head. We want to set it to <code>replica</code> for postgre 9.6+, for 9.5 <code>hot_standby</code>. We follow the <code>wal_level</code> with a few other settings that are good defaults.</p>
<div class="files">
/etc/postgresql/10/main/postgresql.conf<br />
/etc/postgresql/9.6/main/postgresql.conf<br />
</div>
<div class="highlight"><pre><span></span>listen_addresses = '*'
wal_level = replica # In releases prior to 9.6, this parameter also allowed the values archive and hot_standby. These are mapped to replica.
max_wal_senders = 2
wal_keep_segments = 32
archive_mode = on
archive_command = 'cp -i %p /var/lib/postgresql/10/main/archive/%f' # Use with PostgreSQL 10
archive_command = 'cp -i %p /var/lib/postgresql/9.6/main/archive/%f' # Use with PostgreSQL 9.6
</pre></div>
<div class="files">
/etc/postgresql/9.5/main/postgresql.conf
</div>
<div class="highlight"><pre><span></span>listen_addresses = '*'
wal_level = hot_standby
max_wal_senders = 2
wal_keep_segments = 32
archive_mode = on
archive_command = 'cp -i %p /var/lib/postgresql/9.5/main/archive/%f'
</pre></div>
<p>To me the key to understanding and getting replication working is the <code>archive_command</code> this is where you set the location of the archive files for our replicas to restore from. In the case of our archive folder we have it in the base of the postgres main install folder. This is an ideal location because later when we pull the data it will pull not only the archive but a similar set of configurations as well. That way we get a good configuration and setup for our replica.</p>
<h2 id="make-archive-folder">Make Folder for Saving Archive To<a class="headerlink" href="#make-archive-folder" title="Permalink to this headline"><i class="fa fa-link"></i></a></h2>
<p>So with the last paragraph in mind we need to make the <code>archive</code> folder, and it needs to have the owner be the <code>postgres</code> user. You can either use <code>chown</code> or temporarily change to the <code>postgres</code> user to create the folder.</p>
<div class="highlight"><pre><span></span>sudo su - postgres
mkdir /var/lib/postgresql/10/main/archive
mkdir /var/lib/postgresql/9.6/main/archive
mkdir /var/lib/postgresql/9.5/main/archive
</pre></div>
<h2 id="restart-primary">Restart Primary Server<a class="headerlink" href="#restart-primary" title="Permalink to this headline"><i class="fa fa-link"></i></a></h2>
<p>At this point our primary server is done being configured, and we just need to restart postgres.</p>
<div class="highlight"><pre><span></span>sudo systemctl restart postgresql <span class="c1"># or</span>
sudo service postgresql restart
</pre></div>
<h2 id="replication-server">Replication Server<a class="headerlink" href="#replication-server" title="Permalink to this headline"><i class="fa fa-link"></i></a></h2>
<p>The replication server(s) is as simple as a standard PostgreSQL server with a couple of configuration tweaks. First install postgres like we have done above, and then follow along below. It is fairly similar in that you install, tweak configuration, run a sync command, create a file and restart the server.</p>
<h2 id="configure-replica">Configure Replica postgresql.config<a class="headerlink" href="#configure-replica" title="Permalink to this headline"><i class="fa fa-link"></i></a></h2>
<p>Much like with the primary server just edit the <code>postgresql.conf</code> file and add the following lines to the bottom. This sets up that we are going to pull data from the primary server.</p>
<div class="files">
/etc/postgresql/10/main/postgresql.conf<br />
/etc/postgresql/9.6/main/postgresql.conf
</div>
<div class="highlight"><pre><span></span>wal_keep_segments = 32
wal_level = replica
hot_standby = on
wal_log_hints = on
</pre></div>
<div class="files">
/etc/postgresql/9.5/main/postgresql.conf
</div>
<div class="highlight"><pre><span></span>wal_keep_segments = 32
wal_level = hot_standby
hot_standby = on
wal_log_hints = on
</pre></div>
<h2 id="sync-data">Sync Primary Data to Replica<a class="headerlink" href="#sync-data" title="Permalink to this headline"><i class="fa fa-link"></i></a></h2>
<p>Now that our configuration is set we need to pull over all of our data, and some base files. We are running a <code>main</code> setup for postgres so we need to remove that folder and all of its contents. Then we need to pull the data over from the primary server. The first is a simple <code>rm -rf</code> command. The later uses <code>pg_basebackup</code></p>
<p><code>pg_basebackup</code> is called with the host you want to pull data from. Then the directory we want to pull over. Since we deleted the <code>main</code> folder we need to be sure to pull that one from the primary server. This will pull in the base files, plus all of the files in the <code>archive</code> folder which we will use for restoring to our replica. You also need to do this as the <code>postgres</code> user so that it sets proper permissions.</p>
<div class="highlight"><pre><span></span>sudo su - postgres
rm -rf /var/lib/postgresql/10/main/
rm -rf /var/lib/postgresql/9.6/main/
rm -rf /var/lib/postgresql/9.5/main/
pg_basebackup -h 192.168.1.103 -D /var/lib/postgresql/10/main -P -U replication
pg_basebackup -h 192.168.1.103 -D /var/lib/postgresql/9.6/main -P -U replication
pg_basebackup -h 192.168.1.103 -D /var/lib/postgresql/9.5/main -P -U replication
</pre></div>
<h2 id="recovery-file">Configure recovery.conf for Continuing Replication<a class="headerlink" href="#recovery-file" title="Permalink to this headline"><i class="fa fa-link"></i></a></h2>
<p>At this point we have installed postgres, configured it, and pulled over all of the data. Now we need to add the file that will connect and constantly pull data over. It will also do the restore command of the data. The file needs to be named <code>recovery.conf</code>, note the <code>y</code>, and it needs to go in the <code>main</code> folder we pulled over. This file needs to be added after a sync, and does not belong on the primary server.</p>
<p>The contents set the connection info to the primary server, along with password. It also sets the command for restoring files. This works since we are pulling WAL logs. It is just moving them to where postgres will restore them automatically. This file needs to be owned by the <code>postgres</code> user.</p>
<div class="files">
/var/lib/postgresql/10/main/recovery.conf<br />
/var/lib/postgresql/9.6/main/recovery.conf<br />
/var/lib/postgresql/9.5/main/recovery.conf
</div>
<div class="highlight"><pre><span></span>standby_mode = 'on'
primary_conninfo = 'host=192.168.1.103 port=5432 user=replication password=password1'
trigger_file = '/var/tmp/standby.trig'
restore_command = 'cp /var/lib/postgresql/10/main/archive/%f %p'
restore_command = 'cp /var/lib/postgresql/9.6/main/archive/%f %p'
restore_command = 'cp /var/lib/postgresql/9.5/main/archive/%f %p'
</pre></div>
<p>Note the <code>trigger_file</code> if you create that file it will stop the replication process on the replica server.</p>
<h2 id="restart-replica">Restart Replica Server<a class="headerlink" href="#restart-replica" title="Permalink to this headline"><i class="fa fa-link"></i></a></h2>
<p>Now our replica is configured so we just need to restart. After a restart it should start the process of restoreing replicated data, and pullig over new data as well.</p>
<div class="highlight"><pre><span></span>sudo systemctl restart postgresql
</pre></div>
<h2 id="test-replication">Test Replciation is Working<a class="headerlink" href="#test-replication" title="Permalink to this headline"><i class="fa fa-link"></i></a></h2>
<p>Your database server should be replicating, but lets test it to be sure. It is as simple as creating a new database on the primary server, then logging in on the replica and validating it is there.</p>
<h3>On Primary</h3>
<div class="highlight"><pre><span></span>sudo su - postgres
psql
create database test1;
</pre></div>
<h3>On Replica</h3>
<div class="highlight"><pre><span></span>sudo su - postgres
psql
\l
</pre></div>
<h2 id="final-thoughts">Final Thoughts<a class="headerlink" href="#final-thoughts" title="Permalink to this headline"><i class="fa fa-link"></i></a></h2>
<p>This is just the tip of the iceberg with regards to what you can do with replication of postgresql, hopefully though it gives you a jumping off point. This is great for creating a primary server to write to and a replica you can read from so you can distribute some of the load of using a database. Another great thing about the read-only replica is it is ideal for setting up your database backups to pull from there, again reducing load on the primary.</p>
<p>At this point you should be good to go. The next thing I suggest is taking a look at <code>pg-pool</code> for doing connection pooling between both of the servers. It is a great tool, and I hope to come out with a guide on it soon as well.</p>Quick and Dirty Write Your Own Bash Autocomplete2016-09-16T10:00:00-05:002018-02-02T10:00:00-06:00Buddy Lindseytag:buddylindsey.com,2016-09-16:/quick-and-dirty-write-your-own-bash-autocomplete.html<p>Typing a command on the command line and hitting tab to get the rest of what you need is very nice. It is almost magical. Fortunately, it is quite easy to write your own...</p><p>Typing a command on the command line and hitting tab to get the rest of what you need is very nice. It is almost magical. Fortunately, it is quite easy to write your own script to handle this for any application you have, or write.</p>
<p>This is a quick and dirty intro into how to do this. It is meant to give you a jumping off point to write your own autocompletions not be a comprehensive tutorial</p>
<h2>All the Code You Need.</h2>
<p>To get started there is some code that you just need to add with out really understanding so you can get something done.</p>
<div class="highlight"><pre><span></span>_djangoadmin<span class="o">()</span>
<span class="o">{</span>
<span class="nb">local</span> <span class="nv">cur</span><span class="o">=</span><span class="si">${</span><span class="nv">COMP_WORDS</span><span class="p">[COMP_CWORD]</span><span class="si">}</span>
<span class="nv">COMPREPLY</span><span class="o">=(</span> <span class="k">$(</span><span class="nb">compgen</span> -W <span class="s2">"runserver collectstatic"</span> -- <span class="nv">$cur</span><span class="k">)</span> <span class="o">)</span>
<span class="o">}</span>
<span class="nb">complete</span> -F _djangoadmin django-admin.py
</pre></div>
<ul>
<li>First you are creating a new function called _djangoadmin to be called later.</li>
<li>Line 3 is boilerplate to just add to your function.</li>
<li>Line 4 is where the magic is. After the -W in the quotes is where you put your autocompleted commands.</li>
<li>Line 6 is saying that when you type in django-admin.py and hit tab run the _djangoadmin fuction.</li>
</ul>
<h2>Conclusion</h2>
<p>After seeing this I quickly realized that since it is fairly simple to get started I need to offer an autocomplete script if I ever write a CLI utility for other people to use. It is also useful for writing your own autocomplete for utilities that don’ have it.</p>django request.POST vs request.method2014-01-06T10:00:00-06:002018-02-03T10:00:00-06:00Buddy Lindseytag:buddylindsey.com,2014-01-06:/django-request-post-vs-request-method.html<p>Properly processing requests in your views is important, and one of the most important things is knowing what HTTP verb is being used. There are few ways to determine whether you are getting a <code>GET</code> or <code>POST</code>. The two most common ways people determine this...</p><p>Properly processing requests in your views is important, and one of the most important things is knowing what HTTP verb is being used.</p>
<p>There are few ways to determine whether you are getting a <code>GET</code> or <code>POST</code>. The two most common ways people determine this is:</p>
<div class="highlight"><pre><span></span><span class="k">if</span> <span class="n">request</span><span class="o">.</span><span class="n">POST</span><span class="p">:</span>
<span class="k">pass</span>
</pre></div>
<p>or</p>
<div class="highlight"><pre><span></span><span class="k">if</span> <span class="n">request</span><span class="o">.</span><span class="n">method</span> <span class="o">==</span> <span class="s1">'POST'</span><span class="p">:</span>
<span class="k">pass</span>
</pre></div>
<p>It is important to know the difference between the two. While they do get you to the same place there are a couple of caveats to note.</p>
<h2>request.method</h2>
<p>This actually returns a string of the method used with the request, and nothing else. This is important because you can use HTTP verbs without sending data.</p>
<h2>request.POST</h2>
<p>If you do a boolean check of request.POST it checks to make sure that there is data in the <code>POST</code> <code>QueryDict</code> dictionary like object. If there is data then it was a <code>POST</code>; if no data then it evaluates as false as if no <code>POST</code> happened.</p>
<p>The problem is you can do a <code>POST</code> even without data and if you do you would get the following result: (captured from the shell)</p>
<div class="highlight"><pre><span></span><span class="n">request</span><span class="o">.</span><span class="n">POST</span>
<span class="o"><</span><span class="n">QueryDict</span><span class="p">:</span> <span class="p">{}</span><span class="o">></span>
</pre></div>
<p>Which means there is no data so your code says “this is not a <code>POST</code>” when it really is one.</p>
<h2>Conclusion</h2>
<p>If you don’t know how these two different parts of the framework work then it can lead to a lot of headache when you get results you don’t expect. I recommend to always use <code>request.method</code> instead of evaluating if the <code>QueryDict</code> has data in it from a <code>GET</code> or a <code>POST</code>. It can lead to headaches in logic in some few instances, and is more explicit as to what is being evaluated.</p>Quick Intro to Python Requests Library2013-05-07T10:00:00-05:002018-02-02T10:00:00-06:00Buddy Lindseytag:buddylindsey.com,2013-05-07:/quick-intro-to-python-requests-library.html<p>Let's face it using urrlib is hard, and is not really fun at all. Using it to call your own api's or 3rd party api's is an exercise in frustration. Let's take a look at...</p><p>Calling 3rd party services is an essential part of web development these days. I did a quick little python article on <a href="/basic-urllib-get-and-post-with-and-without-data.html">Basic urllib <code>GET</code> and <code>POST</code> With and Without Data</a>. It was a good look into how python natively handles doing <code>GET</code> and <code>POST</code> HTTP actions. However, there is a better way, and that is with the <a href="http://docs.python-requests.org/en/latest/">requests library</a>.</p>
<h2>GET</h2>
<p>Most of what you will be doing is using different HTTP Verbs. <code>GET</code> is probably the one you will use the most, and it is simple to do. Look at the following code example:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">requests</span>
<span class="n">r</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'https://localhost/user/buddylindsey/'</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">text</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">json</span><span class="p">())</span>
</pre></div>
<p>You can even pass in some data with your get request.</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">requests</span>
<span class="n">r</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span>
<span class="s1">'https://localhost/user/'</span><span class="p">,</span>
<span class="n">params</span><span class="o">=</span><span class="p">{</span><span class="s1">'user'</span><span class="p">:</span><span class="s1">'buddy'</span><span class="p">}</span>
<span class="p">)</span>
</pre></div>
<p>This builds up the request and adds the data as a query-string automagically. Makes things a bit easier don’t you think?</p>
<h2>POST</h2>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">requests</span>
<span class="n">r</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">post</span><span class="p">(</span>
<span class="s1">'http://localhost/user/new'</span><span class="p">,</span>
<span class="p">{</span><span class="s1">'username'</span><span class="p">:</span><span class="s1">'buddylindsey'</span><span class="p">,</span> <span class="s1">'password'</span><span class="p">:</span><span class="s1">'password'</span><span class="p">}</span>
<span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">status_code</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">json</span><span class="p">())</span>
</pre></div>
<h2>Other Verbs</h2>
<p><code>OPTIONS</code>, <code>HEAD</code>, <code>PUT</code>, <code>PATCH</code> and <code>DELETE</code> are also available. To be honest I haven’t used <code>OPTIONS</code>, <code>PATCH</code>, or <code>HEAD</code> before so while I have a basic understanding of how they work I won’t attempt to explain it to you. Instead please visit the <a href="http://docs.python-requests.org/en/latest/user/advanced/#http-verbs">requests HTTP verbs</a> docs to get a better understanding of their usage.</p>
<h2>More on Response Object</h2>
<p>The great thing about the response object you get back after making a request is you all of the options available for things to do and know. Here are some of the methods and properties you have access to:</p>
<ul>
<li>headers</li>
<li>status_code</li>
<li>text</li>
<li>json()</li>
<li>encoding</li>
</ul>
<p>There are many more, but these are probably the ones you will use the most.</p>
<h2>Conclusion</h2>
<p>The requests library is one of the best, if not the best, libraries for calling 3rd party web services and acting upon them. This is a powerful tool to have in your toolbelt, it is recommended to learn it, and learn it well.</p>Python Date and Datetime Objects – Getting to Know Them2013-04-09T10:00:00-05:002018-02-02T10:00:00-06:00Buddy Lindseytag:buddylindsey.com,2013-04-09:/python-date-and-datetime-objects-getting-to-know-them.html<p>Properly dealing with dates can be hard, but they don’t have to be as long as you understand the basics. I have had to deal a lot with dates lately with a few Django applications, and life got a whole lot easier once I figured out...</p><p>Properly dealing with dates can be hard, but they don’t have to be as long as you understand the basics. I have had to deal a lot with dates lately with a few Django applications, and life got a whole lot easier once I figured out the basics with python. In this post we are going talk about getting dates, difference between date and datetime, and how to add and subtract time. Timezones I want to discuss in a blog post on its own.</p>
<h2>Datetime Module</h2>
<p>The datetime module is what houses all of our objects/classes for dealing with dates. You will need to import datetime whenever you want to deal with dates. It is super convenient to use and is built right into python. The key classes are:</p>
<ul>
<li>date – Just a date. (Month, Day, Year)</li>
<li>time – Time independent of day. (Hour, Minute, Second, Microsecond)</li>
<li>datetime – Combination of date and time. (Month, Day, Year, Hour, Second, Microsecond)</li>
<li>timedelta – A duration of time used for manipulating dates</li>
<li>tzinfo – An abstract class for dealing with timezones</li>
</ul>
<p><em>NOTE 1:</em> All of these as objects are immutable.<br />
<em>NOTE 2:</em> Objects of type date are naive, meaning they are not aware of a timezone.</p>
<p><strong>Naive</strong> and <strong>Aware</strong> are two states you need to be aware of when dealing with dates. Naive dates just assume there is no timezone, basically. Aware dates you can use to make adjustments as needed. Again we are going to discuss timezone based dates in another post.</p>
<h2>date vs datetime</h2>
<h3>date</h3>
<p>Date just gets you back the actual date, not time. The best/easiset way to get the date is to create a new object and add in the appropriate numbers.</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">date</span>
<span class="n">d</span> <span class="o">=</span> <span class="n">date</span><span class="p">(</span><span class="mi">2013</span><span class="p">,</span> <span class="mi">12</span><span class="p">,</span> <span class="mi">22</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">d</span><span class="o">.</span><span class="n">year</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">d</span><span class="o">.</span><span class="n">month</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">d</span><span class="o">.</span><span class="n">day</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">d</span><span class="o">.</span><span class="n">strftime</span><span class="p">(</span><span class="s2">"%Y %m </span><span class="si">%d</span><span class="s2">"</span><span class="p">))</span>
</pre></div>
<p>This will output:</p>
<div class="highlight"><pre><span></span>>>> <span class="s1">'2013'</span>
>>> <span class="s1">'12'</span>
>>> <span class="s1">'22'</span>
>>> <span class="s1">'2013 12 22'</span>
</pre></div>
<h3>datetime</h3>
<p>Datetime is fairly similar as well, and with some of the same code you can do generate a date and time object to work with.</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">datetime</span>
<span class="n">d</span> <span class="o">=</span> <span class="n">datetime</span><span class="p">(</span><span class="mi">2013</span><span class="p">,</span> <span class="mi">12</span><span class="p">,</span> <span class="mi">22</span><span class="p">,</span> <span class="mi">11</span><span class="p">,</span> <span class="mi">30</span><span class="p">,</span> <span class="mi">59</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">d</span><span class="o">.</span><span class="n">hour</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">d</span><span class="o">.</span><span class="n">minute</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">d</span><span class="o">.</span><span class="n">second</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">d</span><span class="o">.</span><span class="n">strftime</span><span class="p">(</span><span class="s2">"%m/</span><span class="si">%d</span><span class="s2">/%Y %I:%M:%S"</span><span class="p">))</span>
</pre></div>
<p>This will output:</p>
<div class="highlight"><pre><span></span>>>> <span class="m">11</span>
>>> <span class="m">30</span>
>>> <span class="m">59</span>
>>> <span class="m">2</span>/22/2013 <span class="m">11</span>:30:59
</pre></div>
<p>You may have noticed at the end I used the <code>strftime</code> function. This is a convenient function to output the date/time in string format. All you need to do is pass it a string with a symbol and the function interprets it and outputs it properly. <a href="http://strftime.org">Strftime.org</a> is the site I use to get the symbols to add since it looks nicer and is easier to remember than the python docs.</p>
<h2>Manipulating Dates</h2>
<p>The final thing we shall look at is adding and subtracting time. How I used to do it before I actually looked in the docs to find the proper way was manually get the time using hours, minutes, seconds properties and create a new object adding or subtracting appropriately. After getting frustrated I found the <code>timedelta</code> object and it is amazing.</p>
<p>Lets add 3 days to a date object.</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">timedelta</span><span class="p">,</span> <span class="n">date</span>
<span class="n">d1</span> <span class="o">=</span> <span class="n">date</span><span class="p">(</span><span class="mi">2012</span><span class="p">,</span> <span class="mi">12</span><span class="p">,</span> <span class="mi">20</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">d1</span><span class="o">.</span><span class="n">strftime</span><span class="p">(</span><span class="s2">"%Y %m </span><span class="si">%d</span><span class="s2">"</span><span class="p">))</span>
<span class="n">d2</span> <span class="o">=</span> <span class="n">d1</span> <span class="o">+</span> <span class="n">timedelta</span><span class="p">(</span><span class="n">days</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">d2</span><span class="o">.</span><span class="n">strftime</span><span class="p">(</span><span class="s2">"%Y %m </span><span class="si">%d</span><span class="s2">"</span><span class="p">))</span>
</pre></div>
<p>This will output:</p>
<div class="highlight"><pre><span></span>>>> <span class="s1">'2013 12 20'</span>
>>> <span class="s1">'2013 12 23'</span>
</pre></div>
<p>You can try to experiment from here with dealing with timedelta objects. It kind of makes dealing with dates fun and easy.</p>
<h2>Conclusion</h2>
<p>Dealing with dates while programming is a science it itself, an annoying one at times. Fortunately, the python devs have solved the problem, at least as far as I have used it. The great thing about doing dates using python date objects is when adding and subtracting days, years, hours, etc you get a more accurate day and month. Subtracting 53 days from the 14th of March can be a bit complicated if you do it by hand, but with python its super simple. So if you aren’t using date objects in python you should definitely start today.</p>Minimum Grep You Need to Know2013-04-02T10:00:00-05:002018-02-02T10:00:00-06:00Buddy Lindseytag:buddylindsey.com,2013-04-02:/minimum-grep-you-need-to-know.html<p>You need to find a phrase in over 200 files worth of code. Manual searching is not a feasible option. If you are like me you know about <code>grep</code>, but...</p><p>You need to find a phrase in over 200 files worth of code. Manual searching is not a feasible option. If you are like me you know about <code>grep</code>, but it has always made you nervous. It is so powerful and robust that reading the man page was like a tech manual for an engine. Fortunately, getting the benefits of <code>grep</code> with little pain is easy, once you finally figure it out.</p>
<p>Over the last few months I have had to use <code>grep</code> more and more, and I would say I use the same type of search 80% of the time. It gets me what I need quickly and efficiently without much fuss.</p>
<h2>What is Grep</h2>
<p>The best full explanation comes from the <code>grep</code> man page.</p>
<div class="highlight"><pre><span></span>Grep searches the named input FILEs (or standard input if no files are named,
or the file name – is given) for lines containing a match to the given pattern.
By default, grep prints the matching lines.
</pre></div>
<p>My explanation is it finds stuff in files and shows you where it is. It is amazingly useful because of the speed, and by showing the line it found it on along with the file name so you can more easily compare if that is what you need over just a file name.</p>
<h2>Using Grep</h2>
<p>Grep is very powerful, but I get by with 3 variations day-to-day for most of my needs. First lets look at how to structure your <code>grep</code> command.</p>
<div class="highlight"><pre><span></span>grep <options> <search-term> <location>
</pre></div>
<p>This is important to remember as it can be frustrating when you forget and nothing works.</p>
<ul>
<li>options – these are the different flags that can help you get more robust or targeted results back.</li>
<li>search-term – this takes any pattern/regular expression to match against all the files you are searching</li>
<li>location – this is where you put a directory or leave blank to search <code>stdout</code>/<code>stdin</code></li>
</ul>
<h2>Grep with Other Commands</h2>
<p>If you don’t put in a location it searches <code>stdout</code>/<code>stdin</code>. That is useful if you pipe (<code>|</code>) a bunch of data to <code>grep</code> for searching. A mundane example is:</p>
<div class="highlight"><pre><span></span>ls -lha <span class="p">|</span> grep buddy
</pre></div>
<p>This does a normal <code>ls -lha</code> and passes the result to <code>grep</code>. From there it only returns lines that have the word “buddy” in them.</p>
<p>To show the regular expression usage you can do:</p>
<div class="highlight"><pre><span></span>ls -lha <span class="p">|</span> grep ^d
</pre></div>
<p>This returns only results where the line starts with <code>d</code>. In the case of <code>ls</code> it means only directories are returned. As long as you are using the <code>l</code> option which displays all the data.</p>
<h2>Grep’ing Files</h2>
<p>Where you will probably spend most of your time is searching for text inside of files. Mostly you will need to know the file and line number of where the word you are looking for is located.</p>
<div class="highlight"><pre><span></span>grep -rn hello .
</pre></div>
<p>This searches for hello in every file in the current directory and subdirectory. It then shows you the line in the file and the file number it is on. The options are fairly easy to remember as well:</p>
<ul>
<li>r – recursively search files</li>
<li>n – display line numbers</li>
</ul>
<p>Excluding Directories
Sometimes you get too many results or you get results in the folders you don’t want to search in. One of the projects I work on at work has a .svn folder that needs to stay. So I usually have to not include the directory. Fortunately it is easy.</p>
<div class="highlight"><pre><span></span>grep -rn --exclude-dir<span class="o">=</span>.svn hello .
</pre></div>
<h2>Conclusion</h2>
<p>Above is about all you need to know to get started using <code>grep</code>. It is an awesome tool with a lot more features cane you can do some crazy cool searches. It also actually helps you find elusive pieces of code.</p>Basic urllib GET and POST With and Without Data2012-09-05T10:00:00-05:002018-02-03T10:00:00-06:00Buddy Lindseytag:buddylindsey.com,2012-09-05:/basic-urllib-get-and-post-with-and-without-data.html<p>In an age of web services being able to interact with them is almost a necessity for any developer. Unfortunately, if you are a new’ish python developer it might be a bit hard to figure out how to do it since...</p><p>In an age of web services being able to interact with them is almost a necessity for any developer. Unfortunately, if you are a new’ish python developer it might be a bit hard to figure out how to do it since there is no straight forward way. So in this post I will show you some of the basics by doing a <code>GET</code> and <code>POST</code>, and showing the nuances.</p>
<p>This blog post assumes you know the purpose of <code>GET</code> and <code>POST</code> HTTP verbs.</p>
<h2>Getting Acquainted with urllib</h2>
<p>Urllib is a built-in python library which allows you a, fairly, easy way to interact with remote resources. The problem is if you don’t know how to use it, at first, it can be confusing.</p>
<p>There are two main methods you need to be concerned with:</p>
<div class="highlight"><pre><span></span><span class="n">urllib</span><span class="o">.</span><span class="n">urlopen</span><span class="p">()</span>
<span class="n">urllib</span><span class="o">.</span><span class="n">urlencode</span><span class="p">()</span>
</pre></div>
<p><code>urllib.urlopen</code> is what you will actually use to call a web service. While <code>urllib.urlencode</code> is what you are going to use to prepare your data to send to your web service. Those two functions are really all you need to know to get started.</p>
<h2>GET Without Data</h2>
<p>Sometimes all you are needing to do is get data with no special filters or anything so all we need to do is call <code>urlopen</code> and give it a url.</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">urllib</span>
<span class="n">u</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="s2">"http://localhost/users/all/"</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">u</span><span class="o">.</span><span class="n">read</span><span class="p">())</span>
</pre></div>
<p>Lets take a walk through the code:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">urllib</span>
</pre></div>
<p>This imports our library for use. Fairly basic stuff.</p>
<div class="highlight"><pre><span></span><span class="n">u</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="s2">"http://localhost/users/all/"</span><span class="p">)</span>
</pre></div>
<p>Here we are actually calling our web service and assigning an instantiated IO type object similar to python File object. This gives us a few options how to deal with our call, and data. Since we are going to keep this basic we are going to just look at one method.</p>
<div class="highlight"><pre><span></span><span class="k">print</span><span class="p">(</span><span class="n">u</span><span class="o">.</span><span class="n">read</span><span class="p">())</span>
</pre></div>
<p>This is where the fun is since this dumps all the data we got back from our web service to console. Here you would generally apply some sort of deserializer to get the data in a more programmatic format. I usually deal a lot with JSON so you would want to decode the JSON and deal with it from there.</p>
<p>Now that we have an idea of what we are doing lets look at a couple of different variations to how we call because that determines what urllib does.</p>
<h2>GET With Data</h2>
<p>Sometimes we need get data, but it needs a filter on it so we need to send a bit of data along with our get request.</p>
<div class="highlight"><pre><span></span><span class="n">data</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">urlencode</span><span class="p">({</span><span class="s2">"contains"</span><span class="p">:</span><span class="s2">"buddy"</span><span class="p">})</span>
<span class="n">u</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="s2">"http://localhost/users/all/?{}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
</pre></div>
<p>With line one we urlencoded our dictionary which turned the dictionary into a query string type syntax we then just added to the end of our urlopen call. So what urlopen method really called was “http://localhost/users/all/?contains=buddy”. The good thing about doing it the way we did is in case there are special characters which need to be encoded it can take care of it. Also beats the heck out of manually building a long set of query strings.</p>
<h2>POST Data</h2>
<p>The final thing on our list of what we are doing is <code>POST</code>ing data to a web service. This is done the exact same way, except with a few minor character differences. Lets take a look.</p>
<div class="highlight"><pre><span></span><span class="n">data</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">urlencode</span><span class="p">({</span><span class="s2">"name"</span><span class="p">:</span><span class="s2">"buddy lindsey"</span><span class="p">,</span><span class="s2">"height"</span><span class="p">:</span><span class="s2">"5 foot 5 inches"</span><span class="p">})</span>
<span class="n">u</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="s2">"http://localhost/users/create"</span><span class="p">,</span> <span class="n">data</span><span class="p">)</span>
</pre></div>
<p>If you look at the code the only thing that really changed was I added more data, and the <code>urlopen</code> call is a bit different. Instead of adding data as string to the end of the url it is passed as a parameter to the urlopen method. This tells urlopen “hey use a <code>POST</code> instead of a <code>GET</code>”. Nice fun nuance isn’t it.</p>
<h2>Conclusion</h2>
<p>It is amazing how something so simple can be so convoluted when you start. The first time learning it it isn’t very simple so I hope this is laid out well. There is much more to learn to do more advanced “stuff” with web services, but this will get you most of the way there dealing with them, without a 3rd party library.</p>Getting Started with Mixing Fabric Deployments and VirtualEnv2012-08-14T10:00:00-05:002018-02-03T10:00:00-06:00Buddy Lindseytag:buddylindsey.com,2012-08-14:/getting-started-with-mixing-fabric-deployments-and-virtualenv.html<p>Automating your deployment can be one of those things that is tricky to get right at first, but pays off in spades once done. It seems as you develop you continually evolve your processes until you finally figure out how to get your deploys...</p><p>Automating your deployment can be one of those things that is tricky to get right at first, but pays off in spades once done. It seems as you develop you continually evolve your processes until you finally figure out how to get your deploys to be fairly automated. I have been using <a href="https://github.com/fabric/fabric">Fabric</a> for a while now, and it is one of the best tools I have used in a long time.</p>
<p>The problem is there aren’t a lot of great tutorials on how to get going with fabric. Most things I see just drop and script and explain line by line, if your lucky. In this blog post I want to walk you through as if you haven’t ever automated a deploy before and evolve a script before your eyes.</p>
<h2>Prerequisites</h2>
<p>I am not going to walk you through the actual hosting with apache, gunicorn or uwsgi. This guide assumes you have that worked out on your own. We will only discuss deploying to a place on server over SSH.</p>
<h3>Server</h3>
<p>You will need a few things on your server:</p>
<ul>
<li>SSH Access. Fabric uses ssh to push changes</li>
<li>VirtualEnv and a location where you have your Virtual Environments stored</li>
<li>Fabric installed in the global site-packages</li>
</ul>
<h3>Local Machine (or deployment machine)</h3>
<p>You can set a fabric script to be executed automatically or you can run it yourself. The minimum you need is fabric installed locally to run the fabric script.</p>
<ul>
<li>Python</li>
<li>Fabric</li>
<li>Fabric Deploy Script</li>
</ul>
<h2>Fabric Basics</h2>
<p>Fabric can be used to do more than just a deploy. In fact it can be used to do all sorts of remote stuff automagically. All fabric does at its core is execute commands locally and remotely. Fabric just provides a nice pythonic way of doing it.</p>
<h3>Installation</h3>
<p>Fabric is easy to install via <code>pip</code> or <code>easy_install</code></p>
<div class="highlight"><pre><span></span>easy_install fabric
<span class="c1"># or</span>
pip install fabric
</pre></div>
<h3>fabfile.py</h3>
<p>All of you’re fabric scripts need to start in a file called <code>fabfile.py</code>. It is the convention fabric uses so it assumes the file is there when you call the fab command.</p>
<h3>run()</h3>
<p>The only other thing you “need to know” to get started is about the <code>run</code> function. It executes whatever command you put in as a parameter on the server. Here is an example:</p>
<div class="highlight"><pre><span></span><span class="n">run</span><span class="p">(</span><span class="s2">"mkdir buddysite"</span><span class="p">)</span>
</pre></div>
<p>This would create a directory called buddysite in the root folder of what ever user was used with SSH. So if you are using your root account it creates a folder at <code>/root/buddysite</code>, and if your username is buddy it creates <code>/home/buddy/buddysite</code>.</p>
<h3>Set Servers</h3>
<p>The last thing I like to setup in my deployment file is a default server, or set of servers. That is very simple to do using <code>env.hosts</code> at the top. Something like this:</p>
<div class="highlight"><pre><span></span><span class="n">env</span><span class="o">.</span><span class="n">hosts</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'user@server.com'</span><span class="p">]</span>
</pre></div>
<p>That setting will tell the deployment to use the user <code>user</code> and server address <code>server.com</code> it also might look like:</p>
<div class="highlight"><pre><span></span><span class="n">env</span><span class="o">.</span><span class="n">hosts</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'buddy@191.168.42.178'</span><span class="p">]</span>
</pre></div>
<h3>Closing Basics</h3>
<p>So now we have determined we need fabric installed on the server and your local machine. You need a file named <code>fabfile.py</code> and will use the function <code>run</code> to run commands on the remote server you set in the <code>env.hosts</code> setting. So far it is simple stuff lets actually do something a bit more and deploy some code.</p>
<h2>Deploying Code</h2>
<p>First we are going to look at a super basic script that gets our code from our git repository onto our server, and that is it. Then we will refine the script a bit more to deal with <code>VirtualEnv</code> as well.</p>
<h3>The Deploy Script</h3>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">fabric.api</span> <span class="kn">import</span> <span class="o">*</span>
<span class="n">env</span><span class="o">.</span><span class="n">hosts</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'webuser@192.168.42.137'</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">deploy</span><span class="p">():</span>
<span class="n">run</span><span class="p">(</span><span class="s2">"git clone https://github.com/user/repo.git"</span><span class="p">)</span>
</pre></div>
<p>That is really the most simple deployment you can get with fabric. There are problems with it, but really just shows how basic things can get.</p>
<p>Now to actually deploy just do:</p>
<div class="highlight"><pre><span></span>fab deploy
</pre></div>
<h3>Walking Through the Code</h3>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">fabric.api</span> <span class="kn">import</span> <span class="o">*</span>
</pre></div>
<p>As you have probably guessed this imports all the common fabric things we need when running our deployment. We don't generally like useing <code>*</code> import, but we are in this case since as of the writing of this the official docs say to.</p>
<div class="highlight"><pre><span></span><span class="n">env</span><span class="o">.</span><span class="n">hosts</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'webuser@192.168.42.137'</span><span class="p">]</span>
</pre></div>
<p>As discussed before this sets our server and user we are deploying to.</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">deploy</span><span class="p">():</span>
</pre></div>
<p>This is the function that we call when we do our deploy. From the <code>fab deploy</code> command above it just calls the function that is in the <code>fabfile.py</code> file. Deploy is generally what people use as a starting point the great thing is its just a function.</p>
<div class="highlight"><pre><span></span><span class="n">run</span><span class="p">(</span><span class="s2">"git clone https://github.com/user/repo.git"</span><span class="p">)</span>
</pre></div>
<p>This will checkout your code onto your server and since it is just using the ssh user it will be in <code>/home/webuser/repo</code>. Later we need to do better at telling it where to go.</p>
<h3>Adding Specific Locations</h3>
<p>We don’t really want to run stuff from the location of the root of our users directory. We will have conventions on our server of where to deploy stuff so this is a sample of the code specifying a location to put our code when cloning it.</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">fabric.api</span> <span class="kn">import</span> <span class="o">*</span>
<span class="n">env</span><span class="o">.</span><span class="n">hosts</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'webuser@192.168.42.137'</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">deploy</span><span class="p">():</span>
<span class="k">with</span> <span class="n">cd</span><span class="p">(</span><span class="s2">"/some/path/www/"</span><span class="p">):</span>
<span class="n">run</span><span class="p">(</span><span class="s2">"git clone https://github.com/user/repo.git"</span><span class="p">)</span>
</pre></div>
<p>We only made one change to the code</p>
<div class="highlight"><pre><span></span><span class="k">with</span> <span class="n">cd</span><span class="p">(</span><span class="s2">"/some/path/www/"</span><span class="p">):</span>
</pre></div>
<p>Everything in this context manager will be executed in the folder you set. So instead of checking out code to <code>/home/webuser/</code> it will be checked out to <code>/some/path/www/</code>. Just make sure at the beginning you have a <code>/</code> so it starts at the root of the file system.</p>
<h3>VirtualEnv and Package Requirements</h3>
<p>In order to do virtual environments with our script we are going to make a couple of assumptions.</p>
<ul>
<li>The virtual environment is in <code>/home/webuser/venv/mysite/</code></li>
<li>The virtual environment exists already</li>
</ul>
<p>I usually have a bit more complex things going on than this, but it is the easiest way to start off. The next thing we need to do is install our requirements inside of the virtual environment.</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">fabric.api</span> <span class="kn">import</span> <span class="o">*</span>
<span class="n">env</span><span class="o">.</span><span class="n">hosts</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'webuser@192.168.42.137'</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">deploy</span><span class="p">():</span>
<span class="k">with</span> <span class="n">cd</span><span class="p">(</span><span class="s2">"/some/path/www/"</span><span class="p">):</span>
<span class="n">run</span><span class="p">(</span><span class="s2">"git clone https://github.com/user/repo.git repo"</span><span class="p">)</span>
<span class="n">run</span><span class="p">(</span><span class="s2">"source /home/webuser/venv/mysite/bin/activate && pip install -r /some/path/www/repo/requirements.txt"</span><span class="p">)</span>
</pre></div>
<p>If you note the code change:</p>
<div class="highlight"><pre><span></span><span class="n">run</span><span class="p">(</span><span class="s2">"source /home/webuser/venv/mysite/bin/activate && pip install -r /some/path/www/repo/requirements.txt"</span><span class="p">)</span>
</pre></div>
<p>That activates our virtual environment then installs all of our packages to make sure things are going to work when our server uses the virtual environment to execute our site code.</p>
<h3>Closing the Deploy Script</h3>
<p>So now we have a working deploy which puts our code where it needs to go, and even installs our requirements in a virtual environment. We are still running a very basic deployment, and really any more code just makes the deployment easier to maintain or lets more complex scenarios get played out.</p>
<p>I hope you have noticed to this point nothing in this has been django specific so you can use this to do just about anything deployment wise. I even use this on rails projects on occasion, but you could even use this for php projects as well.</p>
<h2>Refactoring the Deploy Script</h2>
<p>So I don’t like such hardcoded stuff I like to add variables and move code around so here is the same script with a bit of stuff moved around.</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">fabric.api</span> <span class="kn">import</span> <span class="o">*</span>
<span class="n">env</span><span class="o">.</span><span class="n">hosts</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'webuser@192.168.42.137'</span><span class="p">]</span>
<span class="n">SITE_ROOT</span> <span class="o">=</span> <span class="s2">"/some/path/www"</span>
<span class="n">VENV_DIR</span> <span class="o">=</span> <span class="s2">"/home/webuser/venv/mysite"</span>
<span class="k">def</span> <span class="nf">_install_dependencies</span><span class="p">(</span><span class="n">site_dir</span><span class="p">):</span>
<span class="n">run</span><span class="p">(</span><span class="s2">"source {}/bin/activate && pip install -r {}/{}/requirements.txt"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
<span class="n">VENV_DIR</span><span class="p">,</span> <span class="n">SITE_ROOT</span><span class="p">,</span> <span class="n">site_dir</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">deploy</span><span class="p">():</span>
<span class="n">site_dir</span> <span class="o">=</span> <span class="s2">"repo"</span>
<span class="k">with</span> <span class="n">cd</span><span class="p">(</span><span class="n">SITE_ROOT</span><span class="p">):</span>
<span class="n">run</span><span class="p">(</span><span class="s2">"git clone https://github.com/user/repo.git"</span><span class="p">)</span>
<span class="n">_install_dependencies</span><span class="p">(</span><span class="n">site_dir</span><span class="p">)</span>
</pre></div>
<p>If you look the script still does the same thing just everything is moved around a bit. Please take a look and you will probably see a few things that confuse you, but I put them there to have fun with the next section.</p>
<h2>Adding Very Basic Rollback Ability</h2>
<p>This isn’t going to be necessarily the best way to do this, but it is a way to do it.</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">datetime</span>
<span class="kn">from</span> <span class="nn">fabric.api</span> <span class="kn">import</span> <span class="o">*</span>
<span class="n">env</span><span class="o">.</span><span class="n">hosts</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'webuser@192.168.42.137'</span><span class="p">]</span>
<span class="n">SITE_ROOT</span> <span class="o">=</span> <span class="s2">"/some/path/www"</span>
<span class="n">VENV_DIR</span> <span class="o">=</span> <span class="s2">"/home/webuser/venv/mysite"</span>
<span class="k">def</span> <span class="nf">_install_dependencies</span><span class="p">(</span><span class="n">code_dir</span><span class="p">):</span>
<span class="n">run</span><span class="p">(</span><span class="s2">"source {}/bin/activate && pip install -r {}/{}/requirements.txt"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
<span class="n">VENV_DIR</span><span class="p">,</span> <span class="n">SITE_ROOT</span><span class="p">,</span> <span class="n">site_dir</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">_link_to_current</span><span class="p">(</span><span class="n">code_dir</span><span class="p">):</span>
<span class="c1"># assumes site_root/current exists</span>
<span class="n">run</span><span class="p">(</span><span class="s2">"rm {}/current"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">SITE_ROOT</span><span class="p">))</span>
<span class="n">run</span><span class="p">(</span><span class="s2">"ln -s {} {}/current"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">code_dir</span><span class="p">,</span> <span class="n">SITE_ROOT</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">deploy</span><span class="p">():</span>
<span class="n">temp_dir</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span><span class="o">.</span><span class="n">strftime</span><span class="p">(</span><span class="s2">"%Y%m</span><span class="si">%d</span><span class="s2">%H%M"</span><span class="p">)</span>
<span class="n">code_dir</span> <span class="o">=</span> <span class="s2">"{}/{}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">SITE_ROOT</span><span class="p">,</span> <span class="n">temp_dir</span><span class="p">)</span>
<span class="n">run</span><span class="p">(</span><span class="s2">"mkdir {}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">code_dir</span><span class="p">))</span>
<span class="n">run</span><span class="p">(</span><span class="s2">"git clone https://github.com/user/repo.git {}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">code_dir</span><span class="p">))</span>
<span class="n">_install_dependencies</span><span class="p">(</span><span class="n">code_dir</span><span class="p">)</span>
<span class="n">_link_to_current</span><span class="p">(</span><span class="n">code_dir</span><span class="p">)</span>
</pre></div>
<p>I am not going to go line by line, but what I have done is modify it so that each deploy gets set to its own timestamped folder. From there it symlinks to the <code>current</code> folder. Your web server needs to run the django, or any wsgi based, application out of the folder <code>current</code>. Now you can have a history of deploys so if the latest one breaks you can just symlink to the previous one and things should work again.</p>
<h2>Conclusion</h2>
<p>There are a couple of things I didn’t go over and that is doing migrations, along with resetting the wsgi application execution. Both I will leave to you based on your server configuration, and application specifics. What I do is set them in separate functions and have a <code>big_deploy</code> function that will call the database functionality as well. As for resetting the wsgi execution that depends on your server configuration.</p>
<p>Hopefully this will get you started with using fabric to do a deployment so that you don’t have to manually move things around. I am also sure you can figure out other things to do with this as well. Have fun deploying code now so you can do it more often. As an extra step look at git hooks and play with continuous deployment.</p>Starting an HTML 5 Tile Based Canvas Game – Game Area and Character2012-07-02T10:00:00-05:002018-02-03T10:00:00-06:00Buddy Lindseytag:buddylindsey.com,2012-07-02:/starting-an-html-5-tile-based-canvas-game-game-area-and-character.html<p>One of the most common games people want to make game wise is a tile based RPG. To start you need to add a character to the canvas and be able to move him around with your keyboard that is the primary goal of this article.</p><p>One of the most common games people want to make game wise is a tile based RPG. To start you need to add a character to the canvas and be able to move him around with your keyboard that is the primary goal of this article.</p>
<p>To see this working visit the <a href="http://buddylindsey.github.com/javascript-goofing/game/">github page</a>. Also you can view all of the code in the <a href="https://github.com/buddylindsey/javascript-goofing/tree/master/game">github repository</a>.</p>
<h2>Getting Started</h2>
<p>So to actually get something going we need to start with some basics that allow for everything to work and for future development. Here is what we are going to need:</p>
<ul>
<li>Main Page</li>
<li>Game Area (canvas)</li>
<li>Game Loop</li>
<li>Character</li>
<li>Keyboard Code</li>
<li>Game Engine Execution</li>
</ul>
<h2>Main Page</h2>
<p>The main page is easy to do because it is just some basic html.</p>
<div class="highlight"><pre><span></span><span class="cp"><!DOCTYPE html></span>
<span class="p"><</span><span class="nt">html</span><span class="p">></span>
<span class="p"><</span><span class="nt">head</span><span class="p">></span>
<span class="p"><</span><span class="nt">meta</span> <span class="na">charset</span><span class="o">=</span><span class="s">"utf-8"</span> <span class="p">/></span>
<span class="p"><</span><span class="nt">title</span><span class="p">></</span><span class="nt">title</span><span class="p">></span>
<span class="p"><</span><span class="nt">script</span> <span class="na">src</span><span class="o">=</span><span class="s">"js/helper.js"</span><span class="p">></</span><span class="nt">script</span><span class="p">></span>
<span class="p"><</span><span class="nt">script</span> <span class="na">src</span><span class="o">=</span><span class="s">"js/character.js"</span><span class="p">></</span><span class="nt">script</span><span class="p">></span>
<span class="p"><</span><span class="nt">script</span> <span class="na">src</span><span class="o">=</span><span class="s">"js/game.js"</span><span class="p">></</span><span class="nt">script</span><span class="p">></span>
<span class="p"><</span><span class="nt">script</span> <span class="na">src</span><span class="o">=</span><span class="s">"js/main.js"</span><span class="p">></</span><span class="nt">script</span><span class="p">></span>
<span class="p"></</span><span class="nt">head</span><span class="p">></span>
<span class="p"><</span><span class="nt">body</span><span class="p">></span>
<span class="p"><</span><span class="nt">canvas</span> <span class="na">id</span><span class="o">=</span><span class="s">"gamearea"</span> <span class="na">width</span><span class="o">=</span><span class="s">"480px"</span> <span class="na">height</span><span class="o">=</span><span class="s">"320px"</span><span class="p">></</span><span class="nt">canvas</span><span class="p">></span>
<span class="p"></</span><span class="nt">body</span><span class="p">></span>
<span class="p"></</span><span class="nt">html</span><span class="p">></span>
</pre></div>
<p>The code above is fairly self explanatory so I won’t bore you explaining it (take note of canvas element). However, here is a list of the JavaScript files and their use:</p>
<ul>
<li><em>helper.js</em> – for the key presses</li>
<li><em>character.js</em> – your main object code for the main character of the game</li>
<li><em>game.js</em> – essentially the game engine</li>
<li><em>main.js</em> – the execution of the the rest of the game.</li>
</ul>
<h2>Game Area (Canvas)</h2>
<div class="highlight"><pre><span></span><span class="kd">var</span> <span class="nx">Game</span> <span class="o">=</span> <span class="p">{};</span>
<span class="nx">Game</span><span class="p">.</span><span class="nx">fps</span> <span class="o">=</span> <span class="mi">30</span><span class="p">;</span>
<span class="nx">Game</span><span class="p">.</span><span class="nx">initialize</span> <span class="o">=</span> <span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="nx">entities</span> <span class="o">=</span> <span class="p">[];</span>
<span class="k">this</span><span class="p">.</span><span class="nx">context</span> <span class="o">=</span> <span class="nb">document</span><span class="p">.</span><span class="nx">getElementById</span><span class="p">(</span><span class="s2">"gamearea"</span><span class="p">).</span><span class="nx">getContext</span><span class="p">(</span><span class="s2">"2d"</span><span class="p">);</span>
<span class="p">};</span>
</pre></div>
<p>This sets up our Game object and constructor. The constructor gets the canvas 2d context for us to use in the rest of the engine. The canvas context is where we do all of our work. When we move, rotate, or draw stuff on the screen it all takes place through the canvas context.</p>
<p>We also have an entities array to reference all of our assets that we display in the game. This is a good solution for now since we won’t have that many entities to display. Keep it light is the best solution, for this example.</p>
<div class="highlight"><pre><span></span><span class="nx">Game</span><span class="p">.</span><span class="nx">draw</span> <span class="o">=</span> <span class="kd">function</span><span class="p">(){</span>
<span class="k">this</span><span class="p">.</span><span class="nx">context</span><span class="p">.</span><span class="nx">clearRect</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">480</span><span class="p">,</span><span class="mi">320</span><span class="p">);</span>
<span class="k">for</span><span class="p">(</span><span class="kd">var</span> <span class="nx">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="nx">i</span> <span class="o"><</span> <span class="k">this</span><span class="p">.</span><span class="nx">entities</span><span class="p">.</span><span class="nx">length</span><span class="p">;</span> <span class="nx">i</span><span class="o">++</span><span class="p">){</span>
<span class="k">this</span><span class="p">.</span><span class="nx">entities</span><span class="p">[</span><span class="nx">i</span><span class="p">].</span><span class="nx">draw</span><span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nx">context</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">};</span>
</pre></div>
<p>This is our draw function where we will draw out all of our entities on to our canvas, and this draw function will be called from our game loop which we will hit on later.</p>
<p>First, it clears the canvas of everything and then redraws all of the entities. This is done to consistently draw stuff as things move across the screen or change.</p>
<p>Next we loop through all of our entities and call the draw function which will actually do the drawing of each entity.</p>
<div class="highlight"><pre><span></span><span class="nx">Game</span><span class="p">.</span><span class="nx">update</span> <span class="o">=</span> <span class="kd">function</span><span class="p">(){</span>
<span class="k">this</span><span class="p">.</span><span class="nx">entities</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="nx">update</span><span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nx">context</span><span class="p">);</span>
<span class="p">};</span>
</pre></div>
<p>This update method is what we will use to call updates on everything in the game. For now we are just updating the first entity since we are only planning to have one. It just calls the update method of the entity, and it is up to the entity to know what to do from there.</p>
<div class="highlight"><pre><span></span><span class="nx">Game</span><span class="p">.</span><span class="nx">addPlayer</span> <span class="o">=</span> <span class="kd">function</span><span class="p">(</span><span class="nx">x</span><span class="p">,</span> <span class="nx">y</span><span class="p">){</span>
<span class="nx">Game</span><span class="p">.</span><span class="nx">entities</span><span class="p">.</span><span class="nx">push</span><span class="p">(</span><span class="k">new</span> <span class="nx">Character</span><span class="p">(</span><span class="nx">x</span><span class="p">,</span> <span class="nx">y</span><span class="p">));</span>
<span class="p">};</span>
</pre></div>
<p>This is a very dumbed down way to add characters to the game, but it works for our purposes. Since entities are in an array we are simply pushing a new instantiated object on to the top of the array for our game engine to run.</p>
<h2>Game Loop</h2>
<p>Here is the code for the game loop itself. I wont explain it, but instead link you to the article I got it from which has a better explanation than I can give. Definitely give it a read since it explains a few things specific to doing a game loop in JavaScript.</p>
<ul>
<li><a href="http://nokarma.org/2011/02/02/javascript-game-development-the-game-loop/index.html">Javascript Game Development – The Game Loop </a></li>
</ul>
<div class="highlight"><pre><span></span><span class="nx">Game</span><span class="p">.</span><span class="nx">run</span> <span class="o">=</span> <span class="p">(</span><span class="kd">function</span><span class="p">(){</span>
<span class="kd">var</span> <span class="nx">loops</span> <span class="o">=</span> <span class="mi">0</span><span class="p">,</span> <span class="nx">skipTicks</span> <span class="o">=</span> <span class="mi">1000</span> <span class="o">/</span> <span class="nx">Game</span><span class="p">.</span><span class="nx">fps</span><span class="p">,</span>
<span class="nx">maxFrameSkip</span> <span class="o">=</span> <span class="mi">10</span><span class="p">,</span>
<span class="nx">nextGameTick</span> <span class="o">=</span> <span class="p">(</span><span class="k">new</span> <span class="nb">Date</span><span class="p">).</span><span class="nx">getTime</span><span class="p">();</span>
<span class="k">return</span> <span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="nx">loops</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="k">while</span> <span class="p">((</span><span class="k">new</span> <span class="nb">Date</span><span class="p">).</span><span class="nx">getTime</span><span class="p">()</span> <span class="o">></span> <span class="nx">nextGameTick</span> <span class="o">&&</span> <span class="nx">loops</span> <span class="o"><</span> <span class="nx">maxFrameSkip</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">Game</span><span class="p">.</span><span class="nx">update</span><span class="p">();</span>
<span class="nx">nextGameTick</span> <span class="o">+=</span> <span class="nx">skipTicks</span><span class="p">;</span>
<span class="nx">loops</span><span class="o">++</span><span class="p">;</span>
<span class="p">}</span>
<span class="nx">Game</span><span class="p">.</span><span class="nx">draw</span><span class="p">();</span>
<span class="p">};</span>
<span class="p">})();</span>
</pre></div>
<h2>Character</h2>
<p>Next up is character.js where we have all the code for our main character; creation, drawing, and movement.</p>
<div class="highlight"><pre><span></span><span class="kd">function</span> <span class="nx">Character</span><span class="p">(</span><span class="nx">x</span><span class="p">,</span> <span class="nx">y</span><span class="p">){</span>
<span class="k">this</span><span class="p">.</span><span class="nx">x</span> <span class="o">=</span> <span class="nx">x</span><span class="p">;</span>
<span class="k">this</span><span class="p">.</span><span class="nx">y</span> <span class="o">=</span> <span class="nx">y</span><span class="p">;</span>
<span class="k">this</span><span class="p">.</span><span class="nx">dude</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">Image</span><span class="p">();</span>
<span class="k">this</span><span class="p">.</span><span class="nx">dude</span><span class="p">.</span><span class="nx">src</span> <span class="o">=</span> <span class="s2">"images/person.png"</span><span class="p">;</span>
<span class="p">}</span>
</pre></div>
<p>This is setting up our Character object and the constructor. The constructor is setting the current location and instantiating new Image object for use in displaying our character, along with setting the location of the image. We will use all of this in our draw function, and the coordinates in our update function.</p>
<div class="highlight"><pre><span></span><span class="nx">Character</span><span class="p">.</span><span class="nx">prototype</span><span class="p">.</span><span class="nx">draw</span> <span class="o">=</span> <span class="kd">function</span><span class="p">(</span><span class="nx">context</span><span class="p">){</span>
<span class="k">if</span><span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nx">dude</span><span class="p">.</span><span class="nx">complete</span><span class="p">){</span>
<span class="nx">context</span><span class="p">.</span><span class="nx">drawImage</span><span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nx">dude</span><span class="p">,</span> <span class="k">this</span><span class="p">.</span><span class="nx">x</span><span class="p">,</span> <span class="k">this</span><span class="p">.</span><span class="nx">y</span><span class="p">);</span>
<span class="p">};</span>
<span class="p">};</span>
</pre></div>
<p>All of our entities that will be displayed on the canvas will have a draw function associated with them, and will be called from the game engines draw function. In the case of our function here we check if the image has completed being loaded onto the screen, and if it has then we draw it onto the canvas. It is important to note that we should only draw the image onto the canvas after it has loaded, and that is what the complete property checks.</p>
<div class="highlight"><pre><span></span><span class="nx">Character</span><span class="p">.</span><span class="nx">prototype</span><span class="p">.</span><span class="nx">update</span> <span class="o">=</span> <span class="kd">function</span><span class="p">(</span><span class="nx">context</span><span class="p">){</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">Key</span><span class="p">.</span><span class="nx">isDown</span><span class="p">(</span><span class="nx">Key</span><span class="p">.</span><span class="nx">UP</span><span class="p">))</span> <span class="k">this</span><span class="p">.</span><span class="nx">moveUp</span><span class="p">();</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">Key</span><span class="p">.</span><span class="nx">isDown</span><span class="p">(</span><span class="nx">Key</span><span class="p">.</span><span class="nx">LEFT</span><span class="p">))</span> <span class="k">this</span><span class="p">.</span><span class="nx">moveLeft</span><span class="p">();</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">Key</span><span class="p">.</span><span class="nx">isDown</span><span class="p">(</span><span class="nx">Key</span><span class="p">.</span><span class="nx">DOWN</span><span class="p">))</span> <span class="k">this</span><span class="p">.</span><span class="nx">moveDown</span><span class="p">();</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">Key</span><span class="p">.</span><span class="nx">isDown</span><span class="p">(</span><span class="nx">Key</span><span class="p">.</span><span class="nx">RIGHT</span><span class="p">))</span> <span class="k">this</span><span class="p">.</span><span class="nx">moveRight</span><span class="p">();</span>
<span class="p">}</span>
</pre></div>
<p>So far with our character the only thing we need to do when we update the character is move the character. So all we have to do is check for a keypress and call the proper function from there. We are calling a function from a helper which will be show here in a bit. The update for the character is where we check for the button press since the key press directly deals with character.</p>
<p>As you probably have noticed we are calling move methods from the current object. The following code is what we use for that. All it does is change the x,y coordinates by a bit each time you press the button.</p>
<div class="highlight"><pre><span></span><span class="nx">Character</span><span class="p">.</span><span class="nx">prototype</span><span class="p">.</span><span class="nx">moveUp</span> <span class="o">=</span> <span class="kd">function</span><span class="p">(){</span>
<span class="k">this</span><span class="p">.</span><span class="nx">y</span> <span class="o">-=</span> <span class="mi">2</span><span class="p">;</span>
<span class="p">};</span>
<span class="nx">Character</span><span class="p">.</span><span class="nx">prototype</span><span class="p">.</span><span class="nx">moveDown</span> <span class="o">=</span> <span class="kd">function</span><span class="p">(){</span>
<span class="k">this</span><span class="p">.</span><span class="nx">y</span> <span class="o">+=</span> <span class="mi">2</span><span class="p">;</span>
<span class="p">};</span>
<span class="nx">Character</span><span class="p">.</span><span class="nx">prototype</span><span class="p">.</span><span class="nx">moveLeft</span> <span class="o">=</span> <span class="kd">function</span><span class="p">(){</span>
<span class="k">this</span><span class="p">.</span><span class="nx">x</span> <span class="o">-=</span> <span class="mi">2</span><span class="p">;</span>
<span class="p">};</span>
<span class="nx">Character</span><span class="p">.</span><span class="nx">prototype</span><span class="p">.</span><span class="nx">moveRight</span> <span class="o">=</span> <span class="kd">function</span><span class="p">(){</span>
<span class="k">this</span><span class="p">.</span><span class="nx">x</span> <span class="o">+=</span> <span class="mi">2</span><span class="p">;</span>
<span class="p">};</span>
</pre></div>
<h2>Keyboard Code</h2>
<p>In the helper.js file we have our key press helper to make life easier when dealing with our keyboard, and events dealing with interaction with our game.</p>
<p>I got this code from this article <a href="http://web.archive.org/web/20161028020115/http://nokarma.org/2011/02/27/javascript-game-development-keyboard-input/index.html">Javascript Game Development – Keyboard Input</a>. It is a great helper that helps take events and register actions. The numbers represent keys on the keyboard. In this case the arrow keys.</p>
<div class="highlight"><pre><span></span><span class="kd">var</span> <span class="nx">Key</span> <span class="o">=</span> <span class="p">{</span>
<span class="nx">_pressed</span><span class="o">:</span> <span class="p">{},</span>
<span class="nx">LEFT</span><span class="o">:</span> <span class="mi">37</span><span class="p">,</span>
<span class="nx">UP</span><span class="o">:</span> <span class="mi">38</span><span class="p">,</span>
<span class="nx">RIGHT</span><span class="o">:</span> <span class="mi">39</span><span class="p">,</span>
<span class="nx">DOWN</span><span class="o">:</span> <span class="mi">40</span><span class="p">,</span>
<span class="nx">isDown</span><span class="o">:</span> <span class="kd">function</span><span class="p">(</span><span class="nx">keyCode</span><span class="p">){</span>
<span class="k">return</span> <span class="k">this</span><span class="p">.</span><span class="nx">_pressed</span><span class="p">[</span><span class="nx">keyCode</span><span class="p">];</span>
<span class="p">},</span>
<span class="nx">onKeydown</span><span class="o">:</span> <span class="kd">function</span><span class="p">(</span><span class="nx">event</span><span class="p">){</span>
<span class="k">this</span><span class="p">.</span><span class="nx">_pressed</span><span class="p">[</span><span class="nx">event</span><span class="p">.</span><span class="nx">keyCode</span><span class="p">]</span> <span class="o">=</span> <span class="kc">true</span><span class="p">;</span>
<span class="p">},</span>
<span class="nx">onKeyup</span><span class="o">:</span> <span class="kd">function</span><span class="p">(</span><span class="nx">event</span><span class="p">){</span>
<span class="k">delete</span> <span class="k">this</span><span class="p">.</span><span class="nx">_pressed</span><span class="p">[</span><span class="nx">event</span><span class="p">.</span><span class="nx">keyCode</span><span class="p">];</span>
<span class="p">}</span>
<span class="p">};</span>
<span class="nb">window</span><span class="p">.</span><span class="nx">addEventListener</span><span class="p">(</span><span class="s1">'keyup'</span><span class="p">,</span> <span class="kd">function</span><span class="p">(</span><span class="nx">event</span><span class="p">)</span> <span class="p">{</span> <span class="nx">Key</span><span class="p">.</span><span class="nx">onKeyup</span><span class="p">(</span><span class="nx">event</span><span class="p">);</span> <span class="p">},</span> <span class="kc">false</span><span class="p">);</span>
<span class="nb">window</span><span class="p">.</span><span class="nx">addEventListener</span><span class="p">(</span><span class="s1">'keydown'</span><span class="p">,</span> <span class="kd">function</span><span class="p">(</span><span class="nx">event</span><span class="p">)</span> <span class="p">{</span> <span class="nx">Key</span><span class="p">.</span><span class="nx">onKeydown</span><span class="p">(</span><span class="nx">event</span><span class="p">);</span> <span class="p">},</span> <span class="kc">false</span><span class="p">);</span>
</pre></div>
<h2>Game Engine Execution</h2>
<p>Finally, we are on to actually running our code and moving our character. We have already done all the hard work. All that is left is to now is to run the code.</p>
<div class="highlight"><pre><span></span><span class="nb">window</span><span class="p">.</span><span class="nx">onload</span> <span class="o">=</span> <span class="kd">function</span><span class="p">(){</span>
<span class="nx">Game</span><span class="p">.</span><span class="nx">initialize</span><span class="p">();</span>
<span class="nx">Game</span><span class="p">.</span><span class="nx">addPlayer</span><span class="p">(</span><span class="mi">30</span><span class="p">,</span> <span class="mi">30</span><span class="p">);</span>
<span class="nx">Game</span><span class="p">.</span><span class="nx">_intervalId</span> <span class="o">=</span> <span class="nx">setInterval</span><span class="p">(</span><span class="nx">Game</span><span class="p">.</span><span class="nx">run</span><span class="p">,</span> <span class="mi">0</span><span class="p">);</span>
<span class="p">};</span>
</pre></div>
<p>This is very simple we initialize the game where we do our setup of our game pieces. Then we add our player to the game engine, and set our game to run continuously. That is it, all that is left now is to try it out.</p>
<h2>Conclusion</h2>
<p>The goal with this was to draw a character on the screen and move him around the playing area. It has been a fun adventure learning to do this, and a great dive into canvas without using a library of some sort. After writing all this code I noticed it is fairly simple, but learning it in the first place was kind of difficult. Canvas is big, convoluted, and not well documented on “how” to use it. If you want to get into canvas development be warned it is an uphill battle unless you use a library of some sort, and even then...</p>HTML5 Local Storage Intro2012-01-09T10:00:00-06:002018-02-03T10:00:00-06:00Buddy Lindseytag:buddylindsey.com,2012-01-09:/html5-local-storage-intro.html<p>HTML5 local storage is a really cool feature which is super easy to use and fairly useful, especially for your HTML5 web applications. I have recently started using it for a few items here and there and am impressed with how versatile it is.</p><p>HTML5 local storage is a really cool feature which is super easy to use and fairly useful, especially for your HTML5 web applications. I have recently started using it for a few items here and there and am impressed with how versatile it is.</p>
<h2>Storing Data</h2>
<p>Local storage is a key value pair based storage which stores string data, this is important to note. There really aren’t a lot of built in ways to do data modeling or searching it is just raw data storage. However, it is really cool that it is persistent storage that goes beyond just the current session.</p>
<p>There are a couple of key things to remember about using local storage. It is all based on the browser, computer, and URI you are on. If you change any of those things then you won’t see that stored data anymore.</p>
<h2>Coding With Local Storage</h2>
<h3>Save to Local Storage</h3>
<div class="highlight"><pre><span></span><span class="c1">// Explicit way</span>
<span class="nx">localStorage</span><span class="p">.</span><span class="nx">setItem</span><span class="p">(</span><span class="s2">"link"</span><span class="p">,</span> <span class="s2">"http://buddylindsey.com"</span><span class="p">);</span>
<span class="c1">// Shortcut way</span>
<span class="nx">localStorage</span><span class="p">.</span><span class="nx">link</span> <span class="o">=</span> <span class="s2">"http://buddylindsey.com"</span><span class="p">;</span>
</pre></div>
<p>The first way is fairly easy to figure out. However, the shortcut needs a bit of explaining. Note how I used link at the end of <code>localStorage</code> that is the shortcut. You can change it to just about anything and it will be stored using that key.</p>
<h3>Access Data from Local Storage</h3>
<div class="highlight"><pre><span></span><span class="c1">// Explicity way</span>
<span class="kd">var</span> <span class="nx">data</span> <span class="o">=</span> <span class="nx">localStorage</span><span class="p">.</span><span class="nx">getItem</span><span class="p">(</span><span class="s2">"link"</span><span class="p">);</span>
<span class="c1">// Shortcut way</span>
<span class="kd">var</span> <span class="nx">data</span> <span class="o">=</span> <span class="nx">localStorage</span><span class="p">.</span><span class="nx">link</span>
</pre></div>
<p>Note again on using the key at the end of localStorage to get the data.</p>
<h2>Example Usage</h2>
<p>I have two HTML 5 applications on github which use local storage if you want to look at them and how it is used in a more real world situation. Look in the <code>app.js</code> file.</p>
<ul>
<li><a href="https://github.com/buddylindsey/html5-c25k">html5-c25k</a></li>
<li><a href="https://github.com/buddylindsey/html5-bookmarking">html5-bookmarking</a></li>
</ul>
<p>The <code>html5-c25k</code> app is an example of an actual HTML5 application which can be run as a website <a href="http://buddylindsey.github.com/html5-c25k/">Couch to 5k</a> it is also used as an application with Mozilla’s open web apps (App store now defunct). I even packaged it up in phonegap and it will be submitted to the app store for iOS (Was never approved). It is cool to see how versatile using HTML5 can really be for cross platform compatibility</p>
<h2>Usage in Other Frameworks</h2>
<p>More and more frameworks are being built to utilize <code>localStorage</code> as a storage medium. Sencha touch is one of them and allows for doing better data modeling for mobile web applications. Also backbone.js has some libraries which allow you to override the sync functionality to use local storage as well.</p>
<h2>Conclusion</h2>
<p>As we move more and more to web applications which act as local web apps local data storage is important. Fortunately it is simple to use and with more and more frameworks making it easier to do life shall be more interesting to see what people use with it.</p>Reviewing 2011 the Year that Changed Everything for Me2012-01-06T10:00:00-06:002018-02-03T10:00:00-06:00Buddy Lindseytag:buddylindsey.com,2012-01-06:/reviewing-2011-the-year-that-changed-everything-for-me.html<p>I can honestly say that over the last year I have gone from a beginnermediate developer to an intermediate developer. This was something that was most definitely hard won, but thoroughly pleasing. I just hope...</p><p>I can honestly say that over the last year I have gone from a beginnermediate developer to an intermediate developer. This was something that was most definitely hard won, but thoroughly pleasing. I just hope the quantity of stuff I have learned in the last year and half will hold through the next year.</p>
<h2>Lets Start at the Beginning</h2>
<p>About 18 months ago I started doing development in ASP.NET MVC on the Asian Live Action Database (Site no longer around ~ 2018 udpate). To be honest it was the first solid idea for a project I have really ever had so it was a great learning experience. There was a lot of new design patterns I learned and became familiar with, along with more advanced features of C#.</p>
<p>I developed a good base of knowledge and really got something going, it was a lot of fun and it pushed my brain. Sometime around there a friend convinced me to get a Mac so I can make mobile apps with him, and get some money, and I think that is where things really took off for me. Not because of the mac, but because of everything else.</p>
<h2>New Computer and New Language/Framework</h2>
<p>After getting a base of knowledge from ASP.NET MVC I was really getting the hang of the MVC design pattern and could tell my programming skills were getting up to par, for the most part. About 2 weeks before my fall semester started I decided I wanted a new application to help me with my Japanese class. On a dare from a friend I decided to try it in Rails. This was a major shift not only was I on a new OS, but using a new language and framework all in a new dev environment.</p>
<h3>Riding the Rails to Vim</h3>
<p>Pretty much over the next several months I dove deeply into Ruby on Rails. I wrote a ton of little apps in Rails and really got familiar with it. I even did a couple of contract things with it, and helped a friend migrate over to it. While I never finished the site I set out to do originally, I realized there were a lot of problems doing it as a back end, site and dropped it.</p>
<p>At some point I saw a video of someone using Vim and doing some amazing things with it. The idea of never using the mouse while coding has always intrigued me, but Vim had always been too big of a learning curve, ever tried to quite vim ;), plus my brain wasn’t ready. After staring to use a new OS, and new programming language I was ready to devote myself to learning Vim, so I started to, and still am.</p>
<h2>Github is Awesome, but I Hate the Pricing</h2>
<p>Along with all the new changes I figured it was time to start learning git as well. I was finding many problems with SVN and I really loved the branching and merging features in git. I signed up for Github and started paying the $7 a month and was fine with that since I had only 2 private projects. One was a rewrite of the Asian Live Action Database and a contract app. Around February I had an accident and had some unexpected Medical expenses even with my insurance covering most of it the bills still hurt. So I had to tighten up my belt which included getting rid of github temporarily.</p>
<p>After some talking with a friend we agreed that $7 was a bit expensive for only 5 repositories. We really wanted all our code under source control in separate repos, and not with an archive repo taking up one of the private repositories. So began the path to sourcehold, which of course would be written in Rails.</p>
<h2>Sourcehold: Success or Failure?</h2>
<p>I spent the next 6 months, starting in March, learning and coding sourcehold. It was a very very fun time and I learned a lot. I can actually break it out into 3 distinct sections.</p>
<h3>Learning</h3>
<p>As a beginnermediate developer up to this point doing something like a source control site similar to github was a doable task, but required quite a bit of learning. I had a cursory knowledge of ssh and git. Along with no knowledge of cloud computing other than theory. I proceeded to read the following books just to get started.</p>
<ul>
<li><a href="http://amzn.to/2EeUmX6">ProGit</a></li>
<li><a href="http://amzn.to/2EBYG06">Pro OpenSSH</a></li>
<li><a href="http://amzn.to/2EgyFWz">Host Your Web Site In The Cloud: Amazon Web Services Made Easy</a></li>
<li><a href="http://amzn.to/2DYl8zC">Distributed Programming with Ruby</a></li>
</ul>
<p>These were all good books and I was able to really dissect them and advance my knowledge. I am not sure I could have done this project without reading these books, and it gave me a new respect for programming/technical books. After reading the books I sat down and played with combining as much of it as I could. So I could see how to get everything working together. It was a fun experiment to learn how to get things working together.</p>
<h3>Development</h3>
<p>Once I read all the books and played around. It was time to start actually coding and designing the site. This is really where things got interesting. As a beginnermediate developer I could roll my own code for features, but using 3rd party libraries or especially api’s were almost impossible.</p>
<p>Through this process I had to use 3rd party libraries so it forced me to learn how to interact with them and start using them. It even got me closer to more advanced topics like creating my own reusable code to add to libraries. My goal is to be able to make libraries at some point which really starts to show you are a more advanced developer, I feel like I am at the cusp of doing this.</p>
<p>Along this process I became very familiar with SSH, Ruby and Command Line. 3 things that are very powerful when used together properly.</p>
<h3>Closing Down</h3>
<p>I was able to get sourcehold launched, running, and people using it. However, due to how it worked the site was resource intensive and I learned a valuable business lesson:</p>
<div class="highlight"><pre><span></span>“Customer growth is slow so manage costs”
</pre></div>
<p>I was stuck in a situation where the lowest cost was still a bit high, but my customer adoption rate was so slow it would not be able to cover the costs quick enough. So after 4 months in service I pulled the plug before I ran completely out of money.</p>
<p>One thing to note about when you create something which helps developers is this phrase:</p>
<div class="highlight"><pre><span></span>I will use it on my next project.
</pre></div>
<p>That is the phrase of death. Make sure your project is so good they need to use it now.</p>
<p><em>Update from 2018 - At this point in my life I was making $15,000 a year working server support part time, and contract work here and there. While doing college full-time. I did not have a lot of money to dedicate. If I did this today I could have incurred the cost of this project longer.</em></p>
<h2>Coworking, TulsaWebDevs, Hackathon followed by Startup Weekend</h2>
<p>In August I met up with <a href="http://groovecoder.com">Luke Crouch</a> a guy that forever changed my life. He invited me to our local web developer user group <a href="http://tulsawebdevs.org">TulsaWebDevs</a>. I met a lot of cool people there, and immediately started learning new things by being able to interact with other developers. It was a whole new world of people, and has stretched my brain immensely.</p>
<p>From the TulsaWebDevs I learned about a local coworking venture that started up for local freelancers, and people whom can work remotely. We meet up every Friday for the day and do our work and interact. It turns into a learning experience and exchange of ideas, as well as a place to get help with problems we are having. As someone that doesn’t work on a dev team It has been invaluable because problems I have pounded my head against the wall on I can just ask on fridays and usually someone can answer it for me.</p>
<p>I learned about Hackathon type events from a newspaper article several years ago. I had wanted to do one for a couple of years, but never had an opportunity. I even thought about starting one until Luke managed to organize one himself. It was an awesome event to meet new people and play with new projects. I am definitely in favor of them. 2 weeks after the Tulsa Hackathon I went to Startup Weekend which was just as fun and got to use a new technology which is now shaping some of my blogging efforts.</p>
<h2>GoDjango and Python</h2>
<p>Over the course of the last few months I took on learning Django and python. It has been a lot of fun to finally find a framework and language which really fits well with me. One of the biggest problems in learning Django though were the lack of video tutorials, so I decided to start <a href="https://godjango.com">GoDjango</a> in a similar vain of RailsCasts where there are weekly updates of 5 to 10 minute videos on doing something with Django.</p>
<h2>Closing Out the Year with Mozilla, JavaScript and HTML5</h2>
<p>I definitely can’t close out this post without talking about Mozilla, JavaScript and HTML5, all three are things I never thought I would pay attention to. Luke actually works for Mozilla, which is really cool, so I have had a lot of opportunity to talk and learn about Mozilla from him. As such I have become a major fan of Mozilla and some of the things they are doing, not just FireFox.</p>
<p>Over the course of the last few months I have come to learn that JavaScript and HTML5 are the future, but can easily start being used now, and should be. As such 2 weeks ago I rolled out my first HTMl5 app called <a href="http://buddylindsey.github.io/html5-c25k/index.html">Couch to 5k</a>. It is based on a running program to get you from sitting on the couch to being able to run 5 kilometers. It is hosted on github as a github page.</p>
<p>The really cool thing is that Mozilla has started up with the Mozilla App store, still in preview, and it was super simple to get it on there since it is based on web technologies. All I had to do was add an app manifest and poof it worked. You can install it yourself as well: Couch to 5k Mozilla App Store (App Store Now Defunct ~ 2018).</p>
<h2>Conclusion</h2>
<p>I have left a few things out, but I tried to hit most of the highlights of what I have done. Please checkout my <a href="https://github.com/buddylindsey/">github profile</a> to see more of what I have done. It has been an exciting year and a half, and I look forward to the next year. We shall see what it holds.</p>
<p><em>Update 2018 - I don't have a 2012 in review post, but I ended up getting married to a wonderful christian woman who is the love of my life. I also ended up starting software development full time in 2012 as well.</em></p>Just Launched GoDjango2011-11-19T10:00:00-06:002018-02-03T10:00:00-06:00Buddy Lindseytag:buddylindsey.com,2011-11-19:/just-launched-godjango.html<p>Recently I was compelled to learn python since it seemed like a lot of different forces were driving me to do so. After learning the syntax basics I figured I would take the next step to start learning Django...</p><p>Recently I was compelled to learn python since it seemed like a lot of different forces were driving me to do so. After learning the syntax basics I figured I would take the next step to start learning Django since it is the predominant web framework for python, and I am a web developer.</p>
<p>One of the things I found particularly disheartening is the severe lack of video tutorials available on the web. I am a huge fan of videos because I learn best visually, almost to a fault. There are a few scattered around here and there, but most are out of date. The fortunate things is Django has some of the best documentation for a project I have ever seen, but at the end of the day it is documentation and sometimes some dots don’t connect well in them.</p>
<p>I make the point about video tutorials because, I guess, I was spoiled in the .NET and the Ruby world where there are videos everywhere. Someone is always doing a new one. Take RailsCasts for example, it has been going strong for a long time now. I feel Django can really benefit from some kind of video tutorials as well.</p>
<p>Therefore I have decided to start <a href="https://godjango.com">GoDjango.com</a>. It is a fun project, partly, because I coded the base site using Django and launched it to Heroku. I then recorded the first video and posted it all in about 30’ish hours. I plan to release a new video every week, maybe two, which is short and sweet doing something with Django or surrounding technologies.</p>
<p>Most of the videos are going to be 5 to 15 minute videos which are there to show you how to do 1 thing really quick to get started. I also plan to follow up with more comprehensive videos that cover in a lot of detail and are upwards of an hour long, those will be premium, but very affordable.</p>
<p>With that please visit <a href="https://godjango.com">GoDjango.com</a> and watch the first video. If you feel so inclined, as well, feel free to comment here or there and leave advice suggestions or comments.</p>Getting Started with SQLite2010-09-22T10:00:00-05:002018-02-03T10:00:00-06:00Buddy Lindseytag:buddylindsey.com,2010-09-22:/getting-started-with-sqlite.html<p>You see a lot about SQLite all over the place, but finding a good place to start with how to use and interact with it is a pain. For a long time I never understood how to create, update, edit, etc. a sqlite Database. It turned out...</p><p>You see a lot about SQLite all over the place, but finding a good place to start with how to use and interact with it is a pain. For a long time I never understood how to create, update, edit, etc. a sqlite Database. It turned out to be easier than I thought. Therefore this a quick getting started. One thing to note is to remember that SQLite is a subset of SQL* most basic commands will work, but you might need to look up the nuances.</p>
<h2>Download SQLite</h2>
<p>If you are on linux or OS X you might already have it. Open up command line and try to type:</p>
<div class="highlight"><pre><span></span>sqlite3 –version
</pre></div>
<p>If you get a version number you are good, jump to next step.</p>
<p>If not you need to visit http://sqlite.org/download.html and get your relevent version. For windows get the “Precompiled Binaries”.</p>
<p>If you want you can go ahead and extract the executable and add the .exe to your path, or just run command-line from that folder.</p>
<h2>Create a Database</h2>
<p>Creating a SQLite database was always the most confusing thing for me what to do, but I found it was pretty simple. Most of the time applications use a sqlite library so this is done for the application, but doing it manually was confusing. If you have SQLite executable installed you simply type:</p>
<div class="highlight"><pre><span></span>sqlite3 test.db
</pre></div>
<p>This will create the db file and get you into console mode ready to edit the database. Usually I like to try to find a GUI to edit the db, but for some reason I can’t find a very nice SQLite GUI that will create a db file so I have had to do it manually. Unfortunately, when I try to create the db file if I don’t create a table before I exit it doesn’t actually create and save the file for use; so I usually create a small little table.</p>
<h2>Create Tables</h2>
<p>This is actually really simple since it is ANSI SQL</p>
<div class="highlight"><pre><span></span><span class="k">create</span> <span class="k">table</span> <span class="n">test_table</span><span class="p">(</span><span class="n">id</span> <span class="nb">INTEGER</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span> <span class="n">name</span> <span class="nb">TEXT</span><span class="p">);</span>
</pre></div>
<ul>
<li>“Don’t forget the semi-colon at the end!”</li>
</ul>
<h2>See Schema</h2>
<p>Use the command line tool to view the design/layout of your database. Use <code>sqlite3.exe db_filename.extension</code> on an existing SQLite db file and you are ready to go editing and viewing the file.</p>
<div class="highlight"><pre><span></span><span class="na">.schema</span>
</pre></div>
<p>This command actually outputs the commands issued for the created tables. So it will look just like the command you input above. This is how you will view your database structure.</p>
<h2>Exit and Save</h2>
<p>Finally comes time to quit. This is quite simple.</p>
<div class="highlight"><pre><span></span><span class="na">.exit</span>
</pre></div>
<p>Now you shouldn’t be stuck trying to close the file for 5 minutes because you don’t know what is going on like I did with the stupid period before the word “exit”, it is like trying to quit vim.</p>
<h2>Conclusion</h2>
<p>SQLite is a really useful and portable database file to use, and was pretty “scary” to figure out when I didn’t know anything about it at all. I was so confused by it I didn’t want to mess with it, or anything relating to it. However, considering more and more things I work on seem to be dealing with SQLite db’s it was time to learn it, most notably are RoR and iPhone/OS development. I also like that it works with NHibernate if needed.</p>
<p>I think if this is all you ever learn about SQLite then this is enough to make you comfortable with SQLite when you run across it in the future.</p>
<p>*As noted by Jay R. Wren in the comments. I originally said it was based on T-SQL. This was my confusion because every one I talk to about SQL always says T-SQL or SQL. I never knew the history behind it. SQLite technically uses a subset of ANSI SQL which was formalized by ANSI in 1986, aka SQL-86/7, it has progressed since then and T-SQL is a Microsoft SQL Server and Sybase variant of SQL, sorry for any confusion on this. For more information feel free to read the wiki article on SQL I truely found it interesting. <a href="http://en.wikipedia.org/wiki/SQL">SQL – Wikipedia</a></p>