.

The Byron Project:

Our goal for this project is to map properties that are listed with Byron and Discovery Bay housing deeds.  We are looking to prove that there might be a discrepancy with the location of the residence and the city name on the deed.  We will then create a cartographic map using the GIS derived information.

The only information we had to begin with was a two year old Microsoft Excel file with a list of addresses in Byron and Discovery Bay (some without street numbers), the city and a ten-digit number (column "A").  At the time we did not know what these numbers stood for.  We assumed they were parcel numbers, but we uncertain. Here is the excel file:

On June 16, I was switched to Project Manager for the Byron Project.  Our first attempt at mapping required looking for US Census Tiger line files.  We needed to determine whether we had adequate data for mapping Byron and Discovery Bay.  We needed a recent streets data and anything else that could be helpful towards the completion of our project.  Possibly because of 9/11 and the risk of terrorism, John Radke Ph.D. had mentioned before hand that many websites which offered free GIS data were either restricting users from accessing certain data or completely shutting their sites altogether.  I had experienced this dilemma while looking for data.  Many sites that were recommended by John were shut down. ESRI’s free data center, for example, was no longer available and directed users to other sites. I began searching for tiger data files through the USGS Bay Area Regional Database http://bard.wr.usgs.gov , which provided different types of free data exclusively for the Bay Area and offered free data such as Digital Elevation Models (DEMs), Digital Line Graphs (DLGs), Digital Raster Graphics (DRGs) and Digital Orthophoto Quadrangles (DOQs):

Unfortunately, data was yet to be provided for all areas and one of these is the Byron Discovery Bay region.  There was only hypsography data available.

.

I also searched the GIS Data Depot http://data.geocomm.com, which offers GIS and Geospatial data:

They had a lot of Tiger file information, but I had trouble accessing the zip files.  The data came with an extension of tar.gz at the end of the files.  After googling the web for removing tar.gz extensions, I decided it was best to find other sites with the data that I needed.  I looked under www.geographynetwork.com with no avail.  As I mentioned previous, I searched under www.esri.com but they didn’t offer their free data site anymore.  However, I followed a link for free Census 2000 TIGER/Line Data at http://www.esri.com/data/download/census2000_tigerline/index.html :

Here I downloaded data layers of streets, hydrography, urban areas, place names and water polygons for Contra Costa County, CA. Usually the data comes as zip files.  After I unzipped the files, I placed them in a folder and proceeded to use ArcMap.

.

.

Using ArcMap:

I began by opening the ArcMap Application and started using ArcMap with "A new empty map" and pressed OK :

.

Here is ArcMap:

.

Before I began, I made sure that I had the necessary toolbars. By clicking on View>Toolbars, I made sure that "Standard" and "Tools" were selected.  These are the basic toolbars that are needed in ArcMap:

.

Now I wanted to analyze the data that I downloaded.  To load these files, I used the "Add" Icon to add data    and the following window pops up:

.

I selected the folder which contained the downloaded data and added all the layers into ArcMap by clicking Add.  The data is added as individual layers for Contra Costa County.  tgr0601lkH is the hydrography layer, tgr0601lkA is the streets data, UA_06013 is the urban areas data, tgr06013wat is the water polygons data and tgr06013pm00 is place names data:

.

Using the magnifying tools , I zoomed into the region of Bryon and Discovery Bay. The layers can be "turned on and off" by selecting and deselecting the check boxes.  Here is Bryon and Discovery Bay:

.

The colors can be changed by double-clicking on the colored lines or colored boxes in the "Layers" box:

.

Below is the streets, hydrography and water polygons layers with better suited colors. The urban area and place name layers did not have any relevant information for our project:

.

I zoomed closer into the town of Bryon to analyze the streets layer.  I also "turned on" the labels option to display the roads name on the map.  I did this by right clicking on the name of the streets layer in the "Layers" box and selected Label Features:

It seems that not all of the streets were labeled and it was not as detailed as I expected.  I wasn't sure if these were all the streets in Byron, since there are just a few of them. 

.

On June 19^th I emailed Darin about the Tiger Files and the streets data that I obtained on the web.  

The first meeting for Byron (with myself as the Byron Project Manager) was conducted on June 23rd.  I informed the class about the streets data and since I was skeptical about the roads data, I decided that parcel data would be a better representation for what we needed to conduct our project.  After the meeting, I began to look for parcel property shapefiles, but with no luck.  Some websites offered parcel data for a fee, but this would not help us since they offered individual parcels and not a whole group of data.  Darin believed that we could obtain parcel data from Contra Costa County by asking directly from a GIS division in the Contra Costa County offices in Martinez.  Around June 26 to July 1, Darin called the demographer for Contra Costa who also works with their GIS division. Since Darin knew the new demographer, we believed we could finally obtain the Contra Costa Parcel data.

We waited a few days and after the fourth of July weekend, we found that the demographer could not release the parcel data information to us. But during the week of July 6^th, I tried to find the parcel data again on my own and called ESRI, emailed the USGS, the Data Depot, the Geographic GIS Community, The GIS Center at Berkeley, John Radke's colleagues and other private/public agencies. I had a couple of responses throughout the week where they told me that they did not hold this information and that we needed to contact the Contra Costa County directly.  I had emailed a person from the GIS data clearinghouse who gave me the contact information of John Cunningham, the Senior Transportation Planner at Contra Costa County and who also made a GIS map for CalWorks.  I emailed him inquiring to obtain GIS shapefile data on Contra Costa County for educational purposes.  He replied on July 14^th saying that he could mail the data to cartography group.

We received the CD on July 22^nd.  We finally got a hold of the data that we needed after more than a month and a half since we began our project. We were thrilled that we could get the project underway.

I began to look at the data at the Wurster Hall computer lab since the PC in the cartography lab had a licensing issue.  Since the PC manager was on vacation, I worked with the files at Wurster Hall.

.

Working with ArcMap and the Parcel data:

Since the files came as zip files, I unzipped the files, placed them in a folder called "contracostacounty" and saved the data on the desktop.  I opened the ArcMap application (same method mentioned before) and added the data layers to ArcMap.  This is what we had:

Here we have a combination of points, lines and areas. I analyzed each layer by "turning on" one layer at a time.  GIS data is displayed two ways: graphically/visually as seen above and as a table which contains "embedded" non-mappable information about the data.  This is often called the attribute table. The attribute table tells the ArcMap user exactly what the data reveals, how long or large the lines or polygons are, footnotes about the data, but overall it is very essential and useful information.  To see the attribute table for any layer, you right click the data layer name in the "Layers" Box and select Open Attribute Table. 

.

The first thing I viewed was the parcel data layer.  Everything seemed okay and usable. I could see how easy it would be to determine which residences had Byron or Discovery Bay Housing deeds because the properties are already outlined for you.  Here is Byron and Discovery Bay Parcels:

Town of Byron:

Discovery Bay:

.

Next, I began to view the other layers. From looking at the attribute tables,  b2_node, centerline_2-1-03, Cadastral_line and BookPage layers did not have much usable information for our project.  They did not contain addresses but polygon sizes and line lengths.  At first, it occurred to me that we may not be able to use the parcel data layer because it too only had polygon size lengths:

None of these numbers matched the numbers on the Byron Excel file.  The questions was, "How can we link the Excel file with the Parcel_poly layer?  Viewing the rest of the data layers, there was a layer called APN_Parcels.  Displaying both the Parcel_poly and APN_Parcels layers looked like this:

It seems like a point indicator for each parcel polygon.  

.

Below is the attribute table of APN_Parcels and a field called "APN" which looks similar to the first column of the Byron Excel file. The attribute table looked like this:

.

Both the APN_Parcels layer and the parcels_poly layer showed similar types of information.  Now I was convinced that the APN number and the first column on the Excel file were parcel numbers.  What I needed to create was a single file that would include both sets of  information so that when I would select a specific parcel polygon, it would automatically show the APN number. (This step will be essential later on in order to link the Excel file with the shapefiles.) However, before I could begin, I needed to find a link between the APN_Parcels and Parcel_poly files which showed similar data graphically and non-graphically.  The best way to do this is by using the identify tool  .  The identify tool lets you view attribute information for a single object.  So, I selected one parcel polygon and then it's corresponding APN point (in the center of the parcel polygon) and looked for something similar.  As you can see below, the "PIN" number was similar.  I tried this same analysis for a couple more parcel polygon and points to see if this similarity held consistent.  The "PIN" proved to be the link for both files:

.

To create a single file with the data from two or more layers, I initiated the "Join" function. This will join groups of data together.  To do this I right clicked the primary layer I wanted all my data to be on.  In this case, it is the Parcels_poly layer, since I wanted to retain the graphical representation of the parcel polygons.  If I chose the APN_parcels layer as the primary layer, the outcome will be reversed and all the attribute data will transfer to the APN_parcels layer and show points and not parcel polygons.  So, I right clicked the Parcels_poly layer and selected Joins and Relates>Joins... A pop-up window shows.  I selected Join attributes from a table.  This means that I want to join attributes from a table (the attributes table) to the selected Parcel_poly file.  Number one asks which field in the Parcel_Poly layer would I want to base the join on. This is the PIN field.  Number two asks from which file would I like to join its attribute table with the Parcel_poly layer.  This would be the APN_Parcels layer.  Number three finally asks which field in the attribute table (from the one you selected in number two) would you like to link with what you chose in number one.  In this case it is PIN.  It is a little confusing with the way they word their sentences, but you get used to what they are asking for with a few practices.  After that, I clicked OK and ArcMap completed the request:

.

To check if the join was successful, I selected the identify tool and selected a parcel polygon.  This is what shows up:

Next I right clicked the name Parcel_poly in the "Layers" box, selected Data>Export data and saved this file as joineddata.shp:

I clicked on OK and selected Yes to add the exported data to the map as a layer.  This new shapefile is the new data file with both the Parcel_poly and APN_Parcels attribute information.  

It is always better to save newly joined layers as separate new data layers to prevent confusion of which types were or were not joined.  It is also a good idea to keep the originals as they were. Parcels_poly still has a join with the APN_Parcels layer. To remove this join, right click the Parcel_poly name in the "Layers" box and select Joins and Relates>Remove Join(s) and select APN_Parcels.  Now you have the original Parcel_poly layer.  Since we have the new joineddata shapefile in the layer, I removed all the other irrelevant and unneeded data layers from the layers box.  To do this, right click the data layer's name and select Remove.  By using the identify tool again and clicking on a parcel polygon, I could see that the attribute information is "cleaned up" and shows all the information from the two previous layers.  This is also another good reason to export joined data layers:

.

The next step was to link the Byron excel file to the joineddata shapefile in order to show which parcels were Byron or Discovery Bay. The problem that I encountered was that the Excel file had ten digit numbers while the parcel data had nine digit numbers.  The plan was to shorten the ten digit number to nine so that the file can link with the parcel shapefile data:

Excel file:   Attribute field in the joindata shapefile:

.

 July 22-23:  I began to try and join the Excel file and the joineddata shapefile together, finding that I had a difficult time formatting the Excel file. Since the numbers had dashes, I used the "Find" command in excel to find all "-" and selected Replace All with a blank space.  The problem was that by removing all the dashes, it also removed the first two zeros at the beginning of the number.  In order to join properly, the numbers have to be the same in both files. I tried using the "Format Cells" option in Excel from changing the column from Number to General or Text, et cetera, but that did not work:

..........

.

In the meantime, I wanted to test the theory that the first few columns in the Excel file were similar numbers in the joineddata shapefile. So I  manually formatted a few hundred to see if the join worked if we formatted the rest of the numbers. This was done by basically retyping the numbers in the column as text.  Clearly, the joins worked and there were parcels that has names of Byron and Discovery attached to them. (Details on how to join an Excel file in ArcGIS later).

On July 24^th, we had a class meeting. We discussed what data was available to us from the Contra Costa County department and the issues with formatting the excel file.  One student, Beth, who knew more about Microsoft Excel tried to help with the problem, except that custom formatting the cells did not work well when we were going to do the next step: changing the excel file into a dbfIV file.  ArcGIS cannot "read" an Excel file.  The only recognized extension for ArcGIS is a dbf file. Also on this day, ArcMap in the cartography lab room continued to have a licensing issue and I could not show the class actual GIS maps of Byron and Discovery Bay.  I continued to work in Wurster Hall.

On July 31^st, Don helped us with the Excel file issue. We took the Byron Excel file and opened the file using BBedit.  We then used a similar command of "Find and Replace" all the "-" to blank spaces.  The difference with BBedit and the Excel application is that in BBedit, the numbers will retain the same value/characters when editing the numbers.  The numbers in BBedit were treated like text and when the dashes were removed, the two zeroes in the beginning of the number remained. 

The next step was to remove the last digit of the numbers in the excel file.  Don and Beth helped with this type of number formatting.  After saving the BBedit file and importing it back as an Excel file in Microsoft Excel, they used the "Paste Function" tool .  They selected the function MID from the Text category.  In the pop up window, they entered the following:

The first field indicates from which field to which field Excel will apply the formula.  In this case it is from A2 to A4821.  The second field specifies the starting total digits in that number and the last field specifies the results or how many digits will it end up with.  In this case it is from 10 digits to 9 digits.  Then I pressed OK. 

During this time I joined the Excel file with the joineddata shapefile, using the same method I previously used to join the attribute tables of the Parcels_poly layer and the APN_parcels layer.  This is describe in more detail later in this document. We clearly saw a few scattered homes in Byron and Discovery Bay that were named for another location.  In a class meeting, we decided that before we could continue further into the project, we needed an updated Excel file of Byron and Discovery Bay parcel information.

On August 4^th, Darin contacted Sandi Fredrickson on whether she could obtain the data for us... but I believe Darin obtaines free data on his own through some research.

August 7^th:  Darin obtain the Excel file of recent parcel data and information on a region of Brentwood that would be named Discovery Bay as well.  We agreed that this recent bit of information could be an interesting addition our project, so we included this data in our files. With the same number formatting of the removal of dashes and ten to nine digit numbers, I applied the same procedure that Don and Beth took to display the double zeroes and remove the last digit.  Here is the new Byron/Discovery Bay/Brentwood Excel file:

.

The next step was to join the new Byron/DiscoBay/Brentwood Excel file with the joineddata shapefile. To do this I first needed to save the Excel file as a dbfIV file.  Dr. John Radke had mentioned that there is a glitch with Microsoft Excel, since saving the file and changing it's extension would save all the information in the previous file with the rows and columns extending to infinity.  That is why in order to change an excel file to a dbfIV file, you must copy only the fields with the values you need and paste them to a new document and save it as a dbfIV file.  Once I had this dbfIV file (saved as bydisbrent.dbf), I opened the ArcMap application and added the joineddata layer and the dbfIV file to ArcMap:

.

As for the previous join, I right clicked the name joineddata in he layers box (indicating that this is the primary data for the join), selected Joins and Relates>Joins... and the same window pops up. This time I want to join the joineddata layer with the bydisbrent.dbf file and link both of them by the APN number:

.

After clicking OK, I went to create a new layer.  As in the previous join, I right clicked the joineddata layer, selected Data>Export data, created a new name for the file calling it bydisbrent.shp and added it as a new layer onto ArcMAP.

.

I right clicked the name of the shapefile bydisbrent and selected Properties and selecting the tab Symbology> Categories> Unique Values.  The symbology is very useful in showing a particular aspect of the data layer.  In this case we want to know which parcels have Byron, Discovery Bay or Brentwood location names to them.  In the Value Field, I scrolled down and selected Location, since in the Excel file, the column "LOCATION" showed the names Byron, Discovery Bay and Brentwood.  Next I clicked on Add All Values.  I clicked on the colored rectangles to change the fill color.  Then I clicked OK:

.

Here's what it looks like:

.

Using the magnify tool, I zoomed into the town of Byron:

Things seem to be uniform here.

.

Next I zoom into the region of Discovery Bay:

You can see here that some parcels are scattered about.

.

Next, I wanted to add some street data and water polygon data to this map.  The street and water data will also be very useful for when we make a cartographic map. So I added the street and water data layers that I originally downloaded.  Here are the bydisbrent shapefile and the roads and water data layers in ArcMap:

It seems that they do not line up.  So I check the projects of both the tiger files and the parcel polygon layer.  I right clicked each layer name and selected Properties>Source tab and the projection information shows:

bydisbrent: 

Both tiger files:

.

By looking at both source data, the tiger files do not have a clearly set projection.  It is "Assumed Geographic".  So I proceeded to change the projection of the tiger files.  To do this, I needed to use another application called "ArcToolBox"  . (Before changing projections, the file should not be in use by another application such as ArcMap). This application is useful in manipulating data and changing the data in other extensions or formats.  By opening ArcToolBox, I selected Data Management Tools> Projections>Define Projection Wizard (shapefiles, geodatabase):

.

Before I can project the map, I must define it's projection.

.

Here I select the roads data first and clicked Next:

.

In the next window, I clicked on Select Coordinate System>Select>Projected Coordinate Systems>State Plane>NAD 1983(Feet)>NAD 1983 StatePlane California III FIPS 0403 (Feet).prj and clicked ADD and OK.

.

Here is the output and click Next:

.

The next step is to project the data layer.  Again, using ArcToolBox, I selected Data Management Tools> Projections>Project Wizard (shapefiles, geodatabase).  I selected a tiger file and specified a new output (naming the new file).  I'll call this streets.  Similarly as above, I selected the same projection by clicking on Select Coordinate System>Select>Projected Coordinate Systems>State Plane>NAD 1983(Feet)>NAD 1983 StatePlane California III FIPS 0403 (Feet).prj and clicked and OK.

I opened the ArcMap application once more and added the streets data. Usually this would work but I found that it made things a lot worse.  I changed the symbology of the streets data (width size from 1.00 to 10.00) so that the location of both files are easier to see.  It seemed that changing the tiger file projection to have the same projection as the parcel shapefile made things worse.  Graphically, the tiger files would have been best represented with the "assumed geographic" projection:

.

I tried to fix this problem.  I knew that both files were displayed with the same projection... but the file is shifted somehow. I encountered this in a previous project on projections.  So I analyzed the source file for the tiger file and compared it with the source file of the bydisbrent shapefile.  Everything seemed to be the same except for the extent:

.

So I proceeded to manually change the "extent" of the tiger file. I opened ArcToolBox again and selected the Project Wizard function.  I selected the original streets data (with the defined projection) and went through the same procedure as listed above.  This time when I arrived at the pop-up window describing coordinate extents, I entered the same extent values of bydisbrent. The precision automatically showed the same value as the bydisbrent shapefile:

.

I attempted to project the data layer in ArcMap, only to find that it did not work and the extent information did not change.  It became very frustrating.  So, I resorted to an alternative method of repositioning layers.  Defining and projecting a layer based on a type of projection is the standard way of projecting data.  I would usually be disinclined to manually shift layers around because of the possibility that I would distort the data... but at this time I wanted to try everything. So I used a function called, "Spatial Adjustment" in ArcMap.  To do this I had to open two new toolbars: Editor and Spatial Adjustment.  This can be accessed from View>Toolbars.  To do any modifications on any data layer, one must be in "Editing" mode.  I clicked on Editor and selected Start editing and the destination of the file I want to work with.  With Spatial Adjustment, I chose Set Adjust Data and selected, All features in these layers and pressed OK:

.

This makes both the editor and the the spatial adjustment toolbars active.

So, I began using the displacement tool .  Since both layers are far from each other, there will be a lot of shifting from viewing one layer with another layer and zooming into the layer itself to get an exact displacement point. This might be hard to understand by reading it here.  If both layers were viewable together, creating adjustments would be a lot easier and quicker.

Since ArcMap seems to only allow the adjustment of the parcel shapefile, we will move the parcel shapefile to align with the streets data, instead of the other way around.  So the displacement tool works as two clicks of a beginning point to an ending point. This is called a displacement link. Normally three to five displacement links are best. So starting in the bydisbrent layer I zoomed into a section which would be easier for me to align multiple points.  In this case, that would be in Discovery Bay. I would click my first displacement point at, for example, an intersection.  Then I right click the street layer name of the tiger file and selected Zoom to Layer (to view the layer instead of scrolling to it) and clicked on the magnify tool to zoom closely at the same Discovery Bay intersection.  Once zoomed in, I finally click on the displacement tool function and complete the displacement link by clicking on the exact location of the intersection on the roads data layer.  This creates an arrow point from the starting location to the ending location. To create a few more displacement links, I had to go back to the bydisbrent layer.  I right clicked the bydisbrent name and clicked Zoom to layer.  Then I used the magnifying tool to zoom in to that same Discovery Bay location and picked another intersection.  Then I repeat the same procedure to view the streets data layer and complete the displacement link. I created four displacement links:

Beginning points:

Ending Points:

The layers are so far apart that when we want to see them in relative distance to one another, they are too small to even see:

.

Then I clicked Spatial Adjustment>Adjust to perform the action.  After a few minutes the roads data shifted to my desired location. This is when I finally realized that the roads data did not "fit" the parcel data.  Below you can see that the displacement links tried to place both files together at the intersections.  Only these areas seemed to "fit".  As I zoomed out to see if the fit was uniform throughout, I noticed that both files did not match:

Zooming out:

As I zoomed farther out, I found that the bydisbrent shapefile was also stretched to "fit" the roads data. The lines in bydisbrent are supposed to be perfectly vertical and horizontal just like the streets lines and not shifted like they are here:

I observed that if both files "fit" or aligned together like they should, there would not be a need to tilt one data layer to "fit" another.  If both files aligned very similarly on top of one another, we would not see such a large difference in the alignment.  Unfortunately this means that we cannot use the streets data layer and the cartography group will need to manually add roads to the final map.

Now that the GIS portion of the project was out of the way, the cartography group will create a cartographic map using the data.  This will require the use of MaPublisher and Freehand 10.  MaPublisher will make it possible to use GIS data in the Freehand 10 application.

MaPublisher: (Details using MaPublisher described later in this document)

The bydisbrent shapefile was 67 mb, which isn't a large file to begin with.  I had trouble importing the file in MaPublisher due to memory problems.  After contacting the Avenza company who manufactures MaPublisher, it was recommended that I removed unnecessary attribute information saying that the 67 mb file was an "extremely" large file.  Not thinking that removing attribute information was possible in ArcMap, it didn't occur to me to try this.  So after experimenting in ArcMap, I found out how to discard attribute information.

Procedure:

I opened ArcMap and added the original bydisbrent layer.  I made sure that the "Editor" toolbar was active by selecting View>Toolbars>Editor.  I started editing by selecting Editor>Start Editing. I began by using the "edit" tool .

Here I selected the parcel polygons that I did not need and deleted them.  This takes a few minutes for ArcMap to complete.

.

After I deleted a few parcel polygon around the perimeter of the Byron/Discovery Bay area, I was left with the following:

.

To see if I actually removed data, I opened the attribute table of the bydisbrent shapefile and scrolled through the file.  At the bottom of the screen, it indicates that I have 14157 records or lines of data:

This was a huge difference in comparison of the original file which had 308550 lines of data.

In order to permanently save this data, I selected Editor>Stop Edits>Save Edits.  I renamed this new file bdb.shp for Byron+DiscoveryBay+Brentwood. This new file was 3.9 mb large and an extremely large decrease in mbs from the original file.

Unfortunately the same memory problem occured while trying to import the file into MaPublisher.

Next... Finally importing data into MaPublisher and making the cartographic map/poster. Stay Tuned...