toc

Team LiB
Next Section

The Definitive Guide to MySQL, Second Edition
The Definitive Guide to MySQL, Second Edition
by Michael Kofler ISBN:1590591445
Apress © 2004 (824 pages)

Provides a thorough introduction to the installation, configuration, implementation, and administration of MySQL.

Table of Contents
 | The Definitive Guide to MySQL, Second Edition
 | Preface
 | Part I - Introduction
 | Chapter 1 | - | What Is MySQL?
 | Chapter 2 | - | The Test Environment
 | Chapter 3 | - | Introductory Example (An Opinion Poll)
 | Part II - Fundamentals
 | Chapter 4 | - | User Interfaces
 | Chapter 5 | - | Database Design
 | Chapter 6 | - | An Introduction to SQL
 | Chapter 7 | - | SQL Recipes
 | Chapter 8 | - | InnoDB Tables and Transactions
 | Chapter 9 | - | Security
 | Chapter 10 | - | Administration and Server Configuration
 | Part III - Programming
 | Chapter 11 | - | PHP — Fundamentals and Programming Techniques
 | Chapter 12 | - | PHP — Library Management
 | Chapter 13 | - | PHP — Discussion Forum
 | Chapter 14 | - | Perl
 | Chapter 15 | - | Java, JSP, JDBC
 | Chapter 16 | - | C and C++
 | Chapter 17 | - | Visual Basic, C#, ODBC
 | Part IV - Reference
 | Chapter 18 | - | SQL Reference
 | Chapter 19 | - | MySQL Tools
 | Chapter 20 | - | API Reference
 | Part V - Appendices
 | Appendix A | - | Glossary
 | Appendix B | - | MySQL 4.1
 | Appendix C | - | Example Files
 | Appendix D | - | Bibliography
 | Index
 | List of Figures
 | List of Tables

Team LiB
Next Section


backcover

Team LiB

Back Cover

The Definitive Guide to MySQL, Second Edition, is the perfect all-in-one guide to MySQL. Whether you’re new to Web application development or you’re already an experienced database analyst, this book describes all of the facets of MySQL configuration, administration, and database application development. Greatly expanded from its widely-praised first edition, this updated revision provides additional information about MySQL tools such as MySQL Control Center and phpMyAdmin, database APIs from several popular programming languages available with Linux and Windows and the latest features found in MySQL 4.1.

About the Author

Michael Kofler earned his PH.D. in computer science at Graz Technical University. He has written a number of successful computer books on topics such as Visual Basic, Linux, Mathematica, and Maple. Kofler is also the author of Definitive Guide to Excel VBA, Second Edition, published by Apress.

Team LiB


The Definitive Guide to MySQL, Second Edition

Michael Kofler
Translated by David Kramer

The Definitive Guide to MySQL, Second Edition Copyright © 2004 by Michael Kofler

All rights reserved. No part of this work may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval system, without the prior written permission of the copyright owner and the publisher.

ISBN (pbk): 1-59059-144-5

Trademarked names may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, we use the names only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark.

Translator, Editor, and Compositor:
David Kramer

Editorial Board:
Dan Appleman
Craig Berry
Gary Cornell
Tony Davis
Steven Rycroft
Julian Skinner
Martin Streicher
Jim Sumser
Karen Watterson
Gavin Wray
John Zukowski

Assistant Publisher:
Grace Wong

Production Manager:
Kari Brooks

Proofreader:
Elizabeth Berry

Cover Designer:
Kurt Krames

Manufacturing Manager:
Tom Debolski

Distributed to the book trade in the United States by Springer-Verlag New York, Inc., 175 Fifth Avenue, New York, NY, 10010 and outside the United States by Springer-Verlag GmbH & Co. KG, Tiergartenstr. 17, 69112 Heidelberg, Germany

In the United States: phone 1-800-SPRINGER, email orders@springer-ny.com, or visit http://www.springer-ny.com. Outside the United States: fax +49 6221 345229, email orders@springer.de, or visit http://www.springer.de.

For information on translations, please contact Apress directly at 2560 Ninth Street, Suite 219, Berkeley, CA 94710. Phone 510-549-5930, fax 510-549-5939, email info@apress.com, or visit http://www.apress.com.

The information in this book is distributed on an "as is" basis, without warranty. Although every precaution has been taken in the preparation of this work, neither the author nor Apress shall have any liability to any person or entity with respect to any loss or damage caused or alleged to be caused directly or indirectly by the information contained in this work.

The source code for this book is available to readers at http://www.apress.com, in the Downloads section. You will need to answer questions pertaining to this book in order to successfully download the code.

About the Author

MICHAEL KOFLER EARNED HIS PH.D. in computer science at Graz Technical University. He has written a number of successful computer books on topics such as Visual Basic, Linux, Mathematica, and Maple. Kofler is also the author of Definitive Guide to Excel VBA, Second Edition, published by Apress.

About the Translator

DAVID KRAMER EARNED HIS PH.D. in mathematics at the University of Maryland, and his M.A. in music at Smith College. For many years he worked in academia, first as a professor of mathematics and computer science, and later as a director of academic computing. Over the past eight years he has edited hundreds of books in mathematics and the sciences, and has translated a number of books in a variety of fields, including Definitive Guide to Excel VBA, by Michael Kofler; Enterprise JavaBeans 2.1, by Stefan Denninger and Ingo Peters; and Cryptography in C and C++, by Michael Welschenbach, all published by Apress. Other translations include The Game's Afoot! Game Theory in Myth and Paradox, by Alexander Mehlmann; the children's musical Red Riding! Red Riding!, by Ernst Ekker with music by Sergei Dreznin; In Quest of Tomorrow's Medicines, by Jürgen Drews; and the novel To Err Is Divine, by Ágota Bozai, which will be published by Counterpoint Press in April 2004.


Preface

MYSQL IS THE MOST WIDELY used database system in the Open Source sector. There are many reasons why this is so:

  • MySQL is fast.

  • MySQL is stable.

  • MySQL is easy to learn.

  • MySQL runs on popular operating systems (Windows, Linux, Mac OS X, various flavors of Unix).

  • MySQL applications can be created in a great variety of programming languages (such as C, C++, C#, Java, Perl, PHP, Python, VB, and VB.NET).

  • MySQL is extensively documented on the Internet, and there are many books on the subject available.

  • MySQL is available for many applications free of charge (GPL license).

  • Since the licensing restrictions of GPL are unacceptable for many commercial applications, there are reasonably priced commercial licenses and optional support contracts.

  • MySQL is on the verge of repeating in the database market the success achieved by Linux in the operating system sector. In combination with PHP or Perl, MySQL is providing the database system for more and more web sites. (A favorite combination is Linux + Apache + MySQL + Perl or PHP. Such systems are called "LAMP systems" for short.) MySQL is not just for small web sites; it is used by large firms with huge amounts of data, such as Yahoo!, Slashdot, and Google.

What Does This Book Offer?

This book provides a complete application- and example-oriented introduction to the database system MySQL. No previous knowledge, either of SQL or database design, is assumed.

The introductory Part I of the book begins with an extensive introduction on installation under Windows and Linux. We also consider the installation of components that are used in combination with MySQL (Apache, PHP, phpMyAdmin, Perl, Connector/MyODBC). Building on this, our first example will show the basic use of MySQL and PHP.

Part II, "Fundamentals," provides a large amount of background material on the use of various user interfaces, on the database language SQL, on the proper design of databases, on the use of InnoDB tables, on the access system of MySQL, and on many other topics on administration (such as backups, logging, and replication).

In Part III, "Programming," we emphasize the language PHP: Three chapters cover basic techniques of programming and show how to construct two extensive examples (bookkeeping, a discussion forum). Most of the PHP examples can be tried out live on my web site. We deal with other programming languages, too, with chapters devoted to Perl, Java, C/C++, and VB/VB.NET/C#.

We end the main text with a reference section (Part IV) that provides an overview of the SQL commands of MySQL, the commands and options of the administrative tools, and the functions of important programming interfaces (PHP, Perl, C, Java).

Finally, there are several appendices, comprising (A) a glossary of terms, (B) breaking news on the forthcoming MySQL version 4.1, (C) information on the example files for this book (available at www.apress.com), and (D) a bibliography with suggestions for further reading.

In combination with the example databases and programs, this book should provide a good foundation for the development of your own database applications. In this I wish you much fun and success.

Michael Kofler, August 2003


What Is New in the Second Edition?

There is much that is new. This book has been completely revised, from the first page to the last. (It took as long to produce this second edition as to write the first edition from scratch!) The following list indicates the most important changes:

  • MySQL: The book now covers MySQL 4.0 and to some extent MySQL 4.1. These changes appear throughout the entire book, from installation to the reference section to the appendixes.

  • User interface: This book describes the new user interface MySQL Control Center and the latest version of phpMyAdmin.

  • SQL recipes: There is a new chapter in the introduction on SQL that provides answers to frequently asked SQL questions: How can I process the first/last n data records? How can duplicate records be located? How are subSELECT commands formed in MySQL 4.0 and 4.1? How can a full-text search be accomplished efficiently?

  • InnoDB tables, transactions, rules for integrity: A new chapter is devoted to the InnoDB table format. InnoDB tables enable the execution of transactions with row level locking and the protection of the integrity of relations between tables (foreign key constraints).

  • Security, access rights: Chapter 9 deals with all the innovations dealing with access rights (MySQL privileges). A new section gives a number of practical tips on solving problems with establishing connections.

  • Administration: Everything related to administration is contained in a single clearly structured chapter. New sections deal with the administration of InnoDB tables and server tuning (including Query Cache).

  • PHP: All PHP examples have been reworked in PHP 4.3. The HTML code generated from the PHP examples is now largely XHMTL compatible. The PHP text has been divided over four chapters and is now more clearly structured. In Chapter 11, on techniques of PHP programming, we now also deal with storing images (file upload/download).

  • Java: A new chapter describes programming of Java and JSP applications on the basis of JDBC and Connector/JDBC.

  • C/C++: A new chapter discusses the development of C and C++ programs.

  • VB6: A new example program demonstrates the use of BLOBs.

  • VB.NET, C#: A new section in Chapter 17 deals with the development of ADO.NET applications with the programming languages VB.NET and C#.

  • Reference: The three reference chapters have been thoroughly brought up to date and expanded with new sections (C functions, JDBC classes).

  • MySQL 4.1: Appendix B brings together all known changes in MySQL 4.1: improved character set support (including Unicode), the use of subSELECTs, the management of geometric and geographic data (GIS), the use of precompiled commands (prepared statements), etc.


What Does This Book Not Offer?

Several chapters in this book deal with the programming of MySQL applications in a variety of programming languages, such as PHP, Perl, C, C++, Java, and Visual Basic. These chapters assume that the reader is familiar with the specific programming language. (There is simply no room in this book to offer an introduction to a number of programming languages.) In other words, you will profit from, say, the PHP chapter in this book only if you are already familiar with the programming language PHP.


Example Programs, Source Code

To the extent that we are dealing with web applications, almost all of the applications in this book can be tried out directly on my website (www.kofler.cc). The source code for all the examples is available at www.apress.com in the downloads section.

In the longer program listings in this book you will find at the beginning of the example a comment line that specifies the file name appearing in the example files on the web site, for example,

<!-- php-programming/simpleinput.php -->

To save space, sometimes only the most interesting passages in the program code are printed.


Versions

The functionality of MySQL and of the programs, programming languages, and libraries placed in its environment changes with every new version—which sometimes appear weekly. The following overview indicates which versions I have worked with (explanations of the various names will appear at appropriate places in the book):

  • Apache: Versions 1.3.n and 2.0.n.

  • Connector/J: Version 3.0.6.

  • Connector/ODBC (formerly MyODBC): Version 3.51.06.

  • gcc: Version 3.2

  • Java: Version 1.4.1 (Windows) and 1.3.1 (Linux).

  • JSP: see Tomcat.

  • Linux: MySQL and other programs were tested for this book under Linux and Windows. Under Linux, the distributions Red Hat 8.0 and 9.0 (beta 3) and SuSE 8.1 and 8.2 (beta 5) were used.

  • MySQL: Version 4.0.14. In most of the example programs, however, compatibility with MySQL 3.23.n was taken into account, since this version is still in use by many Internet service providers. On the other hand, the latest developments of MySQL 4.1 were tested with a development version (alpha version, from end of July 2003), as described in Appendix B.

  • Connector/C++: Version 1.7.9.

  • Perl: Versions 5.6 and 5.8.

  • PHP: Versions 4.2 and 4.3. As file identifier, the suffix *.php is generally used.

  • phpMyAdmin: Version 2.4.

  • Tomcat: Version 4.0.4 (Java Servlet 2.2, Java Server 1.1).

  • Visual Basic, VBA, ADO: Visual Basic programs were developed and tested with VB6 and VBA6 and ADO versions 2.1 and 2.7.

  • Visual Basic .NET, C#, ADO.NET, ASP.NET: All of these are based on Visual Studio .NET 2002 (.NET Framework 1.0).

  • Windows: All tests under Windows were carried out with Windows 2000. In principle, everything should be valid for Windows NT 4, Windows XP, and future versions of Windows.


Notation

SQL commands and functions, as well as methods, classes, and key words in SQL, C, Java, PHP, Perl, VB, etc., generally appear in italic (e.g., SELECT, mysql_query). (Note, however, that when the background text is italic, as in the gray "Remark," "Tip," "Caution," and "Pointer" boxes and some section headings, such commands and functions will appear in a roman font; e.g., SELECT.)

  • Unix/Linux user names also appear in italic (e.g., root, mysql).

  • MENU COMMANDS USE CAPS AND SMALL CAPS (e.g., FILE | OPEN).

  • Keyboard shortcuts use a monospace font (e.g., Shift+Delete).

  • File and directory names also use the monospace font (e.g., /usr/local or C:\Windows).

  • Programs and programming commands are in the monospace font as well (e.g., mysql or cmd.exe).

  • MySQL options and configuration parameters appear in regular roman type (e.g., default-charset).

  • Program listings and command line input appear in a sans serif font.

SQL commands are generally written in UPPERCASE letters. This is not a syntactic necessity, but merely a convention. MySQL does not distinguish between uppercase and lowercase in interpreting SQL commands.

In specifying Windows directories, we will often not write out the absolute path, since it depends in any case on the particular installation. We observe the following conventions:

  • \Windows\means the Windows directory (e.g., C:\Windows or D:\WinNT4).

  • Programs\means the directory under Windows for program installation (e.g., C:\Programs or D:\Program Files).

  • Mysql\means the MySQL installation directory (e.g., C:\Mysql or D:\Program Files\Mysql).


Commands

Many commands will be presented in this book. We will be moving back and forth between the Unix/Linux and Windows conventions. The following two commands are equivalent:

root# mysqladmin -u root -h localhost password xxx
> mysqladmin -u root -h localhost password xxx

In each case we have given the system prompt (root# for Unix/Linux and > for Windows). You type in only what follows the prompt (here in boldface type). Under Unix/Linux it is possible to divide long inputs over several lines. The lines are separated by means of the backslash symbol \. We shall often use this convention in this book. The following command thus corresponds to the command above:

root# mysqladmin -u root -h localhost \
            password xxx

In each case, xxx is to be replaced by the relevant text (in this example by your password). We have indicated that xxx is dummy text by the use of a slant font.


Abbreviations

I have attempted in this book to make as little use of abbreviations as possible. However, there are several abbreviations that will be used repeatedly without being introduced anew in each chapter:

ADO | Active Data Objects (Microsoft database library)
BLOB | Binary Large Object (binary data block)
GPL | GNU Public License (important license for Open Source software)
HTML | HyperText Markup Language (format for describing web documents)
InnoDB | Not an abbreviation, but the name of a company that has developed a special table format for MySQL (InnoDB tables)
ISP | Internet Service Provider
JSP | Java Server Pages
MySQL | The name of the company that developed the database system MySQL
ODBC | Open Database Connectivity (interface for database access, particularly popular under Windows)
PHP | PHP Hypertext Preprocessor (a scripting programming language for HTML pages)
RPM | Red Hat Packet Manager (a format for Linux software packages)
SQL | Structured Query Language (database programming language)
URL | Uniform Resource Locator (Internet address of the form http://www.company.com/page.html)
VB | Visual Basic (programming language)
VBA | Visual Basic for Applications (programming language within the Microsoft Office package)


Part I: Introduction

Chapter List

Chapter 1: What Is MySQL?
Chapter 2: The Test Environment
Chapter 3: Introductory Example (An Opinion Poll)


Chapter 1: What Is MySQL?

Overview

This chapter begins with an overview of the most important concepts from the world of databases and then delves into the possibilities and limitations of MySQL. What is MySQL? What can it do, and what is it unable to do?

In addition to describing the central functions of MySQL, we shall also discuss fully the issue of licensing MySQL. When is one permitted to use MySQL without payment, and when is a license required?


What Is a Database?

Before we can answer the central question of this chapter, namely, What is MySQL? you and I must find a common language. Therefore, this section presents a rudimentary database glossary, without going into great detail. (If you have already had significant dealings with relational databases, you can skip the next couple of pages in good conscience.)

There is scarcely to be found a term that is less precise than database. A database can be a list of addresses residing in a spreadsheet program (such as Excel), or it can be the administration files of a telecommunications firm in which several million calls are registered daily, their charges accurately calculated, monthly bills computed, and warning letters sent to those who are in arrears. A simple database can be a stand-alone operation (residing locally on a computer for a single user), while others may be used simultaneously by thousands of users, with the data parceled out among several computers and dozens of hard drives. The size of a database can range from a few kilobytes into the terabytes.[1]

In ordinary usage, the word "database" is used to refer to the actual data, the resulting database files, the database system (such as MySQL or Oracle), or a database client (such as a PHP script or a program written in C++). Thus there arises a great potential for confusion as soon as two people begin to converse on the subject of databases.

Relations, Database Systems, Servers, and Clients

A database is an ordered collection of data, which is normally stored in one or more associated files. The data are structured as tables, where cross references among tables are possible. The existence of such relations among the tables leads to the database being called a relational database.

Let us clarify matters with an example. A database might consist of a table with data on a firm's customers (name, address, etc.), a table with data on the products the firm offers, and finally, a table containing the firm's orders. Through the table of orders it is possible to access the data in the other two tables (for example, via customer and product numbers).

MySQL, Oracle, the Microsoft SQL server, and IBM DB2 are examples of relational database systems. Such a system includes the programs for managing relational databases. Among the tasks of a relational database system are not only the secure storage of data, but also such jobs as the processing of commands for querying, analyzing, and sorting existing data and for storing new data. All of this should be able to take place not only on a single computer, but over a network as well. Instead of a database system we shall often speak of a database server.

Where there are servers, there are clients. Every program that is connected to the database system is called a database client. Database clients have the job of simplifying the use of the database for the end user. No user of a database system in his or her right mind would wish to communicate directly with the database server. That is much too abstract and inconvenient. (Let programmers worry about such direct communication!) Instead, the user has a right to expect convenient tables, listboxes, and so on to enable the location of data or to input new data.

Database clients can assume a variety of forms, and indeed, they are often not recognized by the user as database programs at all. Some examples of this type of client are HTML pages for the display and input of messages in an on-line discussion group, a traditional program with several windows for managing addresses and appointments, and a Perl script for executing administrative tasks. There is thus wide scope for database programming.

Relational Versus Object-Oriented Database Systems

Relational databases have dominated the database world for decades, and they are particularly well suited for business data, which usually lend themselves to structuring in the form of tables. Except for the following two paragraphs, this entire book discusses only relational databases (though we shall not always stress this point).

Another kind of database is the object-oriented database. Such databases can store free-standing objects (without having to arrange them in tables). Although in recent years there has been a trend in the direction of object-oriented programming languages (such as Object-Store, O2, Caché), object-oriented databases have found only a small market niche.

Note that relational databases can be accessed by means of object-oriented programming languages. However, that does not turn a relational database into an object-oriented one. Object-oriented database systems enable direct access to objects defined in the programming language in question and the storage of such objects in the database without conversion (persistency). It is precisely this that is not possible with relational database systems, in which everything must be structured in tables.

Tables, Records, Fields, Queries, SQL, Index, Keys

We have already mentioned tables, which are the structures in which the actual data are located. Every line in such a table is called a data record, or simply record, where the structure of each record is determined by the definition of the table. For example, in a table of addresses every record might contain fields for family name, given name, street, and so on. For every field there are precise conditions on the type of information that can be stored (such as a number in a particular format, or a character string with apredetermined maximum number of characters).

Note

Sometimes, instead of data records with fields, we speak of rows and columns. The meaning is the same.

The description of a database consisting of several tables with all of its fields, relations, and indexes (see below) is called a database model. This model defines the construction of the data structures and at the same time provides the format in which the actual data are to be stored.

Tables usually contain their data in no particular order (more precisely, the order is usually that in which the data have been entered or modified). However, for efficient use of the data it is necessary that from these unordered data a list can be created that is ordered according to one or more criteria. It is frequently useful for such a list to contain only a selection of the data in the table. For example, one could obtain a list of all of one's customers, ordered by ZIP code, who have ordered a rubber ducky within the past twelve months.

To create such a list, one formulates queries. The result of the query is again a table; however, it is one that exists in active memory (RAM) and not on the hard drive.

To formulate a query one uses SQL instructions, which are commands for selecting and extracting data. The abbreviation SQL stands for Structured Query Language, which has become a standard in the formulation of database queries. Needless to say, every producer of a database system offers certain extensions to this standard, which dilutes the goal of compatibility among various database systems.

When tables get large, the speed at which a query can be answered depends significantly on whether there is a suitable index giving the order of the data fields. An index is an auxiliary table that contains only information about the order of the records. An index is also called a key.

An index speeds up access to data, but it has disadvantages as well. First, every index increases the amount of storage on the hard drive necessary for the database file, and second, the index must be updated each time the data are altered, and this costs time. (Thus an index saves time in the reading of data, but it costs time in entering and altering data. It thus depends on the use to which the data are to be put whether an index is on the whole a net plus or minus in the quest for efficiency.)

A special case of an index is a primary index,or primary key,which is distinguished in that the primary index must ensure a unique reference to a record. Often, for this purpose one simply uses a running index number (ID number). Primary indexes play a significant role in relational databases, and they can speed up access to data considerably.

[1]It all started with the megabyte, which is about one million bytes. A terabyte is 1024 gigabytes, which in turn is approximately one thousand megabytes. The prefix "mega-" comes from the Greek for "great," or "large," while "giga-" is derived from the Greek word for "giant." In turn, "tera-" is from the Greek word for "monster." It would appear that numbers once regarded as large, gigantic, or even monstrously huge have become part of our everyday vocabulary.


MySQL

MySQL is a relational database system. If you can believe many diehard MySQL fans, MySQL is faster, more reliable, and cheaper—or, simply put, better—than any other database system (including commercial systems such as Oracle and DB2). Many MySQL opponents continue to challenge this viewpoint, going even so far as to assert that MySQL is not even a relational database system. We can safely say that there is a large bandwidth of opinion.

  • The fact is that there is an ever increasing number of MySQL users, and the overwhelming majority of them are quite satisfied with MySQL. Thus for these users we may say that MySQL is good enough.

  • It is also the fact, however, that MySQL lacks a number of features that are taken for granted with other database systems. If you require such features, then MySQL is (at least for the present) not the database system for you. MySQL is not a panacea.

In this chapter we shall examine some of the possibilities and limitations of MySQL.

Remark

In this book we are considering MySQL. Do not confuse MySQL with mSQL (MiniSQL). To be sure, MySQL and mSQL have similar programming interfaces (APIs), which merely facilitates the confusion between the two. However, the underlying database systems are completely different. More information on mSQL can be found at http://www.hughes.com.au/


Features of MySQL

The following list shows the most important properties of MySQL. This section is directed to the reader who already has some knowledge of relational databases. We will use some terminology from the relational database world without defining our terms exactly. On the other hand, the explanations should make it possible for database novices to understand to some extent what we are talking about.

  • Relational Database System: Like almost all other database systems on the market, MySQL is a relational database system.

  • Client/Server Architecture: MySQL is a client/server system. There is a database server (MySQL) and arbitrarily many clients (application programs), which communicate with the server; that is, they query data, save changes, etc. The clients can run on the same computer as the server or on another computer (communication via a local network or the Internet).

    Almost all of the familiar large database systems (Oracle, Microsoft SQL Server, etc.) are client/server systems. These are in contrast to the file-server systems, which include Microsoft Access, dBase, and FoxPro. The decisive drawback to file-server systems is that when run over a network, they become extremely inefficient as the number of users grows.

  • SQL compatibility: MySQL supports as its database language—as its name suggests—SQL (Structured Query Language). SQL is a standardized language for querying and updating data and for the administration of a database.

    There are several SQL dialects (about as many as there are database systems). MySQL adheres to the ANSI-SQL/92 standard, although with significant restrictions and many a number of extensions.

    This topic will be dealt with more extensively later. Beyond the ANSI-SQL/92 standard, MySQL supports, among other things, several additional data types, full-text indexes, and replication.

  • User Interface: There are a number of convenient user interfaces for administering a MySQL server.

  • Full-text search: Full-text search simplifies and accelerates the search for words that are located within a text field. If you employ MySQL for storing text (such as in an Internet discussion group), you can use full-text search to implement simply an efficient search function.

  • Replication: Replication allows the contents of a database to be copied (replicated) onto a number of computers. In practice, this is done for two reasons: to increase protection against system failure (so that if one computer goes down, another can be put into service) and to improve the speed of database queries.

  • Transactions: In the context of a database system, a transaction means the execution of several database operations as a block. The database system ensures that either all of the operations are correctly executed or none of them. This holds even if in the middle of a transaction there is a power failure, the computer crashes, or some other disaster occurs. Thus, for example, it cannot occur that a sum of money is withdrawn from account A but fails to be deposited in account B due to some type of system error. Transactions also give programmers the possibility of interrupting a series of already executed commands (a sort of revocation). In many situations this leads to a considerable simplification of the programming process.

    In spite of popular opinion, MySQL has supported transactions for a long time. One should note here that MySQL can store tables in a variety of formats. The default table format is called MyISAM, and this format does not support transactions. But there are a number of additional formats that do support transactions. The most popular of these is InnoDB, which will be described in its own chapter.

  • Foreign key constraints: These are rules that ensure that there are no cross references in linked tables that lead to nowhere. MySQL supports foreign key constraints for InnoDB tables.

  • Programming languages: There are quite a number of APIs (application programming interfaces) and libraries for the development of MySQL applications. For client programming you can use, among others, the languages C, C++, Java, Perl, PHP, Python, and Tcl.

  • ODBC: MySQL supports the ODBC interface Connector/ODBC. This allows MySQL to be addressed by all the usual programming languages that run under Microsoft Windows (Delphi, Visual Basic, etc.). The ODBC interface can also be implemented under Unix, though that is seldom necessary.

    Windows programmers who have migrated to Microsoft's new .NET platform can, if they wish, use the ODBC provider or special MySQL providers for .NET.

  • Platform independence: It is not only client applications that run under a variety of operating systems; MySQL itself (that is, the server) can be executed under a number of operating systems. The most important are Apple Macintosh OS X, Linux, Microsoft Windows, and the countless Unix variants, such as AIX, BSDI, FreeBSD, HP-UX, OpenBSD, Net BSD, SGI Iris, and Sun Solaris.

  • Speed: MySQL is considered a very fast database program. This speed has been backed up by a large number of benchmark tests (though such tests—regardless of the source—should be considered with a good dose of skepticism).


Limitations of MySQL

Many of the shortcomings listed in this section can be found on the to-do list of the team of MySQL developers, or have already been implemented.

Pointer

The documentation for MySQL is not at all silent on the subject of shortcomings or missing features. There is a quite readable section in the MySQL documentation on the topic "How standards-compatible is MySQL?" There you will find extensive information on the points at which MySQL fails to comply with current standards. Often, a reason for the shortcoming is provided, and sometimes as well some pointers on how to get around the difficulty:

http://www.mysql.com/doc/en/Compatibility.html

Note also that all of the shortcomings mentioned here are on the to-do list of the developers and will be alleviated in future versions of MySQL.

ANSI-SQL/92 Limitations

ANSI SQL/92 is a standardized definition of the database query language SQL. Many commercial database systems are largely compatible with this standard and also offer many extensions.

MySQL is also conspicuous for its countless extensions, but unfortunately, the compatibility is not so extensive as is the case for other database systems. The points described in greater detail in what follows (subSELECTs, views, triggers, stored procedures) are examples of the lack of ANSI-SQL/92 compatibility.

These limitations usually make it impossible (or merely very difficult) to adapt existing databases and the associated SQL code from other database systems into MySQL. Conversely, it is almost as difficult to transfer a MySQL solution to another database system if you have not scrupulously employed only the ANSI-SQL/92-conforming features of MySQL. (The temptation to use the useful proprietary features proves in most cases to be too great.)

  • Sub SELECT s: MySQL is not capable of executing a query of the form

    SELECT * FROM table1 WHERE x IN (SELECT y FROM table2)

    This limitation can often be circumvented by JOIN operations or by setting up a temporary table. However, that is neither elegant nor particularly efficient. SubSELECTs will soon be supported (from version 4.1; see Appendix B).

  • Views: Simply put, with views we are dealing with an SQL query that is considered an independent database object and that permits a particular view into the database. MySQL currently does not support views (though we may expect this feature to be implemented shortly).

    Of course, with MySQL you can always execute those SQL commands that are defined as views in other databases. However, it is impossible to store these queries in objects in the database.

    Views assist in the administration of the database, since it is thereby relatively simple to control access to individual parts of the database. Views also help in avoiding redundancy in application development.

  • Stored Procedure: When we speak of stored procedures we are referring to SQL code that is stored within a database system. Stored procedures (SPs for short) are usually employed to simplify certain steps in a procedure, such as inserting or deleting a record. For client programmers this has the advantage that these actions need not operate directly on the tables, but can rely on SPs. As with views, SPs also help in the administration of large database projects. Stored procedures can also increase efficiency in certain cases. Stored procedures are expected in MySQL 5.0.

  • Triggers: A trigger is an SQL command that is automatically executed by the server during certain database operations. If a database is incapable on its own of ensuring referential integrity, then it is usually triggers that are brought into play to assist in this. For example, a trigger can be executed every time a record is to be deleted, which would test whether this operation is permissible and prohibit the action if need be.

The lack of subSELECTs, views, SPs, and triggers does not greatly restrict the possibilities open to client programmers, but it does lead to a situation in which the program logic is transferred from the server level to the client level. The result is more complex or expensive client programming than would otherwise be the case, leading to redundancies in code and problems with maintenance and alteration of code.

Further Limitations

The following limitations actually have nothing to do with ANSI-SQL/92 but nonetheless play a significant role in practice.

  • When MySQL is used with standard tables (table type MyISAM), then locking, that is, the temporary blocking of access to or alteration of database information, is in operation only for entire tables (table locking).

    You can circumvent the table-locking problem by implementing transaction-capable table formats, such as InnoDB, that support row locking.

  • In using MyISAM tables, MySQL is not able to execute hot backups, which are backups during operation without blocking the tables with locks. Here again, the solution is InnoDB, though here the hot backup function is available only in the form of a commercial supplement.

  • MySQL 4.0 supports a variety of character sets and the associated sort orders, but at present can have only a single sort order active at a time. If the sort order must be changed, MySQL has to be restarted. All table indexes must then be reset. This limitation has been removed in version 4.1.

  • MySQL 4.0 is unable to deal with Unicode or other mutibyte character strings. (Of course, it is possible to treat such data as binary objects, but there are no functions for dealing with character strings, in particular, no usable sorting or comparison algorithms.) This limitation, too, has been removed in version 4.1.


MySQL Version Numbers

Even for connoisseurs of MySQL it is a bit of a challenge to keep track of which version of MySQL is current and which functions are contained in which versions. This section provides some information on MySQL numbering.

As of March 2002 there are four main versions being worked on by the MySQL development team:

  • MySQL 3.23.n: The first version of this series, 3.23.0, was published in August 1999. Since 3.23.32 (January 2001) MySQL 3.23.n has been considered stable. The current version is 3.23.56.

    MySQL 3.23 is the version most in use among Internet providers, and is only gradually being displaced by version 4.0.n. There will be no further extensions to MySQL 3.23.n.New versions will only fix discovered errors or security holes.

  • MySQL 4.0.n: The first version of this series, 4.0.0, was published in October 2001. Since 4.0.12 (March 2003) MySQL 4.0.n has been considered stable and is now recommended for production use. It is expected that in the coming months, a series of new versions of 4.0.n will appear to resolve errors that have been or are still being discovered. New functionality is not expected in this series.

  • MySQL 4.1.n: The first version in this series was announced in January 2003, but it was not yet available for download as this book was being completed. (Developers can obtain daily updated code from the Internet. However, the MySQL server must itself be compiled from this code, which is no simple matter.) It is not clear when MySQL 4.1.n will be at a point where it can be recommended for production use, probably not before 2004.

  • MySQL 5.0.n: The first published version in this series is expected in 2004. It is completely open as to when the first stable version will be ready. This version is currently of use only to developers who are working on functions that cannot be realized in MySQL 4.n.

Alpha, Beta, Gamma, Production

MySQL versions are identified by the attributes alpha, beta, gamma, and production:

  • Alpha means that the version is in the throes of the development process and that new functions and even incompatible changes are to be expected. Although an alpha version is not published until it contains no known errors, it is highly probable that many undiscovered errors still lurk within. Loss of data during testing of an alpha version is quite possible!

  • Beta means that this version is largely complete, but it has not been thoroughly tested. Major changes are not expected.

  • Gamma means that the beta versions have become more or less stable. The goal now is discover errors and resolve them. Gamma versions are well suited to application developers who want to try out new functions on a test system.

  • Production means that the MySQL development community has the impression that the version is mature and stable enough that it can be used for mission-critical purposes.

According to the MySQL documentation, in production versions, only corrections, and no new functionality, are to be expected. However, this has not always held true in the past. Of particular note is the case of MySQL 3.23.n. After the version had been declared stable (3.23.32), there came general support for InnoDB and BDB tables (3.23.34), and later, integrity rules for InnoDB tables (3.23.44). Furthermore, many minor extensions were introduced. As a rule, MySQL developers are pleased with such extensions, but at the same time, compatibility problems among different production versions can arise.

In practice, this means that a new MySQL version (that is, n.n.0) always has the status alpha. With higher version numbers, the status rises to beta, gamma, and finally, production.

Normal MySQL users should use exclusively MySQL versions that have the status production. If you are developing web applications, you should find out which version your Internet service provider is using. (Since Internet service providers are concernced with maintenance and stability, they are not generally keen on using the latest version, preferring earlier versions, which may not contain many of the functions of the newer versions.)

Tip

If you want to experiment with a development version that is not yet available in compiled format (and thus has not even reached alpha status), instructions on downloading and compiling the source code can be obtained at http://www.mysql.com/doc/en/Installing_source.html

MySQL Functions Ordered by Version Number

It is not always the easiest thing in the world to determine which particular database function can be used with which version of MySQL. Of course, the version number often reveals when a function first became officially available, but it often happens that thereafter, significant changes to the function are made or bugs fixed, so that a stable implementation occurs only several versions later. Note as well that certain functions are available only for particular table formats.

Table 1-1 ventures a glance into the future. The entries are based on information contained in the MySQL documentation as of the end of July 2003, collected from the following web sites:

http://www.mysql.com/doc/en/TODO.html
http://www.mysql.com/doc/en/MySQL_4.0_In_A_Nutshell.html
http://www.mysql.com/doc/en/MySQL_4.1_In_A_Nutshell.html
http://www.mysql.com/doc/en/MySQL-PostgreSQL_features.html
http://www.mysql.com/doc/en/News.html

The projections into the future come, naturally, without warranty of any kind. In the past, it frequently occurred that particular functions became available earlier or later than originally projected.

Table 1-1: MySQL functions past and future
Function | Version
Replication | 3.23
Full-text search in MyISAM tables | 3.23
Transactions with BDB tables | 3.23.34
Transactions with InnoDB tables | 3.23.34
Referential integrity for InnoDB tables | 3.23.44
DELETE across several tables | 4.0
UPDATE across several tables | 4.0
UNION (unite several SELECT results) | 4.0
Query cache (speeds up repeated SQL commands) | 4.0
Embedded MySQL library | 4.0
Encrypted connections (Secure Socket Layer, SSL) | 4.0
Hot backup for InnoDB (commercial add-on product) | 4.0
SubSELECTs | 4.1
Unicode support (UTF8 and UCS2 = UTF16) | 4.1
GIS support (GEOMETRY data type, R-tree index) | 4.1
Prepared Statements (SQL commands with parameters) | 4.1
ROLLUP extension for GROUP BY | 4.1
Stored procedures (SPs) | probably 5.0
Trigger (automatic execution of SQL code) | probably 5.0
Referential integrity for MyISAM tables | probably 5.1
Fail-safe replication | probably 5.1
Hot backup for MyISAM tables | probably 5.1
Column-level constraints | probably 5.1
VIEWs | probably 5.1
XML data type | not currently planned
User-defined data types | not currently planned


MySQL Licensing

One of the most interesting features of MySQL is the license. MySQL is an open source project. That is, the complete source code of MySQL is freely available. Since June 2000 (that is, since version 3.23.19) the GNU Public License (GPL) is valid also for MySQL. It is thus ensured that MySQL will continue to be freely available in the sense of the open source idea. (For commercial applications of MySQL there is a second, commercial, license available in addition to GPL. More on this later.)

Rights and Duties with Respect to the GPL

Open source is often incorrectly interpreted to mean "without cost." It is indeed true that GPL software can be used without payment of fees, provided that one adheres to certain conditions. However, the open source idea goes much further:

  • Since the source code is freely available, when there are problems, you are not at the mercy of a software vendor.

  • When problems arise, you can perhaps attempt to repair the problem yourself or to implement features that are lacking. Furthermore, you can appeal to the developers' group for help.

  • You can be certain that the program code has been read by many developers and does not contain any unsavory surprises (such as so-called back doors such as the database system Interbase had for many years, whereby access to every Interbase database was possible via a hard-coded password).

  • You are permitted to alter GPL products, and indeed sell the resulting new programs.

At the end of this list of GPL merits there are a few demerits (for commercial applications). If you wish to use a GPL program as the basis for a commercial product, you must again make your own source code freely available, in the sense of GPL, with the changes made. This is seldom something that developers of commercial products wish to do.

In general, then, every program that is derived from GPL software exists under the terms of GPL. (GPL is, so to speak, contagious.)

Pointer

Further information on the open source idea, the full text of GPL, and explanations can be found at the following addresses:

http://www.gnu.org/copyleft/gpl.html
http://www.opensource.org/osd.html

Use of MySQL with an Open Source License

The following list collects the different situations in which one may freely use MySQL in the sense of GPL:

  • MySQL can be used without cost if an application is locally developed and not used commercially. It is only when the resulting solution is to be sold to customers that the question of licensing comes into play. This rule is expressed on the MySQL home page as follows: Free use for those who never copy, modify, or distribute.

  • MySQL can be used freely within a web site. If you also develop a PHP application and install it with your Internet service provider, you do not have to make your PHP code freely available in the sense of GPL.

  • Likewise, an Internet service provider may make MySQL available to its customers without having to pay MySQL license fees. (Since MySQL is running exclusively on the ISP computer, this application is considered internal.)

  • Finally, MySQL can be used free of charge for all projects that themselves run under the GPL or comparable free license. (If you have developed a new free e-mail client for Linux, say, and wish to store e-mails in a MySQL database, you may do so without further ado.)

Use of MySQL with a Commercial License

In the sense of GPL the following uses are prohibited:

  • You may not change or extend MySQL (that is, the database server) or sell the new version or product thus created without simultaneously making the source code of your changes freely available. You are thus prohibited from developing a new database system based on MySQL if you are not prepared to make your extensions freely available to the MySQL community in the sense of GPL.

  • It is forbidden to develop a commercial product, such as a bookkeeping program, that is geared toward MySQL as the database without making the code available in the open source sense.

If the limitations of the GPL are not acceptable to you as a commercial developer, then you may sell your product (program) together with a commercial MySQL license. This can prove worthwhile because MySQL remains available to you even if you are unable or unwilling to make your code available in the sense of GPL.

Commercial MySQL licensing fees are calculated based on the number of servers (where a server is considered to be a computer, regardless of the number of CPUs). There is no limitation on the number of clients that may access the server. The cost is quite reasonable in comparison to commercial database systems (currently $220 for a license without InnoDB support and $440 for a license with InnoDB support, with a significant reduction starting at ten licenses).

Pointer

Read the licensing conditions presented in Appendix C! Further information on licensing MySQL can be found in the MySQL documentation, in the chapter "MySQL Licensing and Support," and at the following web sites:

http://www.mysql.com/support/arrangements.html
https://order.mysql.com/

Commercial Licenses for Client Libraries (Connector/ODBC, Connector/J, etc.)

In addition to the actual MySQL server, the MySQL company offers client libraries that are necessary for the development of application programs:

C-API | Application interface for C programming
Connector/C++ | Library for C++ programming
Connector/J | JDBC driver for Java programming
Connector/ODB | ODBC driver for all programming under Windows

Like MySQL, these drivers are available under GPL. Many MySQL client libraries, such as the JDBC driver Connector/J, used to be available in earlier versions under LGPL (lesser Gnu public license), which implies fewer restrictions for commercial applications. The situation has changed, however.

If you wish to use these drivers in commercial applications, the following rule is in force: If MySQL client programs access a MySQL-licensed server, then this server license is valid for the client libraries. It is thus usually unnecessary to obtain licenses for the use of client libraries, because the client libraries are automatically included with the server license.

However, if you develop and sell a commercial program but do not deliver it together with MySQL and leave the installation of the MySQL server and its licensing to the customer, then you must obtain licenses for the client libraries used in your program.

Explicit licensing of the client library is seen by the MySQL company as an exception, since as a rule, the server is licensed. Commercial licensing of the client library actually represents nothing other than a protection of the MySQL company's commercial interests. They want to avoid the situation in which commercial developers get around having to obtain a commercial license simply by not providing the MySQL server. The customer then loads the GPL version of the MySQL server from the Internet and believes that all is right with the world. In such a case the MySQL company would receive nothing for the development of a commercial MySQL product.

MySQL Version Names

Beginning with MySQL 4.0.5, the free (in the sense of GPL) and commercial versions have different names. Here they are:

Name | License | Functionality
MySQL Standard | GPL | All default functions that can be considered stable (including InnoDB)
MySQL Max | GPL | Same as MySQL Standard, but with the BDB table format as well as with additional functions that have not reached full maturity
MySQL Classic | commercial | Like MySQL Standard, but without InnoDB
MySQL Pro | commercial | Like MySQL Standard, but with InnoDB (thus with transactions)

For ordinary GPL applications, the use of the Standard version is recommended. The additional functions in the Max version are for use only in special applications. Currently (version 4.0.12), the Standard and Max versions differ in the support of BDB tables and the SSL encryption of client/server connections. These functions will be incorporated into the Standard and commercial versions only when they are considered sufficiently stable. You can determine which functions the running version of MySQL supports with the following SQL command:

mysql> SHOW VARIABLES LIKE 'have%';

Variable_name | Value
have_bdb | NO
have_crypt | NO
have_innodb | YES
have_isam | YES
have_raid | NO
have_symlink | YES
have_openssl | NO
have_query_cache | YES

In the case of commercial applications, the decision between Classic and Pro depends simply on whether InnoDB is needed. (The license for the Pro version is more expensive.) The Classic and Pro versions cannot be simply downloaded at www.mysql.com, but are provided only after a license has been issued.

If you wish to know which server version is being run, then execute the following SQL command:

mysql> SHOW VARIABLES LIKE 'version';

Variable_name | Value
version | 4.0.12-nt-log

This output shows that the Standard version 4.0.12 is running under Windows NT/2000/XP and that logging is activated. There is no additional Max functionality and no license for commercial use.

Support Contracts

You may make a support contract with the MySQL company if you desire commercial support in the development of a MySQL application. You thereby simultaneously support the further development of MySQL.

Pointer

Details on commercial MySQL licenses and paid support can be found at http://www.mysql.com/support/

Links to various companies that offer commercial MySQL support can be found at http://www.mysql.com/information/partners.html


Alternatives to MySQL

Of course, there are many alternatives to MySQL, particularly if you are prepared to pay (lots of) money for licenses and perhaps also for the requisite hardware. Among these are IBM DB2, Informix, Microsoft SQL Server, and Oracle.

If you are looking for a database in the open source realm, then PostgreSQL is currently perhaps the most interesting alternative. However, be warned: The discussion between advocates of MySQL and those of PostgreSQL usually resembles more a war of religions than what might be termed measured intellectual discourse. In any case, PostgreSQL offers many more functions than MySQL (even if MySQL is slowly catching up). At the same time, it is considered slower and less stable (even if in this regard the reputation of PostgreSQL has greatly improved).

Furthermore, there are several formerly commercial database systems that have been converted to open source. The two best known of these are Firebird (formerly Interbase from Inprise/Borland) and SAP DB.

Pointer

Further information on PostgreSQL, Firebird, and SAP DB can be found on the Internet. The following web pages are a good starting point:


Summary

MySQL is a very capable relational client/server database system. It is sufficiently secure and stable for many applications, and it offers an excellent cost/benefit ratio (not only because MySQL is free itself, but also because it makes comparatively modest demands on hardware). MySQL has thus developed into a quasi standard in the realm of Internet databases.

Above all, in the Linux world, MySQL is used increasingly by applications as the background database engine, whether it be managing logging data more efficiently than previously or managing e-mail, MP3 files, addresses, or comparable data. MySQL is poised to play a similar role to that of the Jet Engine in the Microsoft operating system (where in many respects, MySQL offers a meaningfully better technical basis). Thanks to the ODBC interface, MySQL is now being used in the Windows world for such tasks.

Apart from technical data, MySQL has the advantage over other open source database systems in that it is by far the most widely used such system. It follows that MySQL has been more thoroughly tested and documented than other database systems and that it is relatively easy to find developers with MySQL experience.

However, MySQL cannot (yet) compete in every respect with the big boys of the commercial database system world. You are not likely to choose MySQL if you require functions that MySQL does not yet support.


Chapter 2: The Test Environment

Overview

In this chapter we discuss first of all how to set up the test environment for MySQL on a local computer. Since MySQL usually doesn't run all by itself, but in combination with other programs, this chapter also discusses Apache, PHP, Perl, Connector/ODBC, and phpMyAdmin. We consider installation under Windows and under Linux.

If MySQL has been made available by your Internet service provider (ISP) and you are not inclined to install MySQL on your own computer as well, the last section of this chapter gives a brief overview on how under these conditions you can quickly begin your first experiments.

This chapter is geared specifically to MySQL neophytes. If you already have access to a functioning MySQL system, you may feel free to skip it.


Windows or Unix/Linux?

The question of under which operating system to run MySQL can easily lead to fisticuffs, so strongly are opinions held. But we shall remain civil in this section, and perhaps for the entire book. The fact is that MySQL, as well as Apache, PHP, Perl, and most of the other programs that are usually run in conjunction with MySQL, were developed originally under Unix/Linux and only later ported to Windows.

MySQL in Practice (Public Internet Server)

In practice, that is, on a publicly available server on the Internet, the above-mentioned programs are run predominately under Unix/Linux. (There is scarcely an Internet service provider to be found that offers MySQL under Windows.) For this reason alone—the greater deployment in the world at large—the programs running under Unix/Linux have been better and more extensively tested. Possible errors or security holes in the Unix/Linux version of MySQL are thus more likely to be discovered and repaired quickly.

A further argument for deployment under Unix/Linux is that the programs function more efficiently as a rule. This has less to do with the view that Windows is generally slower than Unix/Linux (I have no desire to discuss that issue here), but with the fact that the process and thread models of these operating systems are quite different from each other. Programs such as Apache and MySQL are first and foremost optimized for the programming model of Unix/Linux, not for that of Windows, and this by itself often gives a significant speed advantage to Unix/Linux.

Therefore, the development communities of the programs mentioned above are unanimously in favor of having their programs run under Unix/Linux, and not under Windows. You would do well to take this opinion to heart.

Remark

When we speak of Windows, we generally mean Windows NT/2000. In principle, MySQL runs under Windows 9x/ME, but there it is impossible to provide reasonable security of the system against unauthorized access. In general, my tests were carried out under Windows 2000.

Development Environment

Things look somewhat different if you are currently only developing a database application. During the development you will use a test environment that is usually accessible only to you or your team. Thus you have no reason to expect problems in security or in efficiency due to large numbers of accesses to the system. Since there is good compatibility between the Windows and Unix/Linux versions, there is little to be said against, for example, developing a discussion group first under Windows and then porting the completed solution to the Linux server of your ISP.

If you have at least two computers in your test environment, you can, of course, install MySQL, Apache, etc., on one computer under Linux, and carry out the actual development work (database design, creation of script files, etc., on the second computer under Windows (say, because your favorite editor runs under Windows).

To this extent, the entire discussion of Windows versus Unix/Linux doesn't amount to much, and for development you should use the operating system that you find more to your liking.

Remark

Of course, there are also arguments against Windows even for the development of MySQL solutions. One is that by developing the entire project under Unix/Linux you gain considerable experience that will be valuable later during the process of bringing the project on line on a Unix/Linux server (specifically in reference toissuesofaccessrights and security).
Remark

Another argument deals with the deployment of system functions that under Unix/Linux usually are available in a form different from that offered under Windows. For example, under Unix/Linux one can simply send an e-mail by program code (such as in a PHP or Perl script). Under Windows there are no standard interfaces for such tasks.

Afinal argument is this: If you possess a typical Linux distribution (for example, SuSE or Red Hat) on CD-ROM, you also possess all the programs of the MySQL test environment. After the various packages have been installed, everything usually runs correctly at first go, without any further configuration nightmares. On the other hand, if you work with Windows, you must first obtain the individual programs from the Internet, install them, and then configure them in what often is painstaking, frustrating labor.


MySQL (Server Installation)

MySQL is available over the Internet without cost, both as source code and in compiled form for most operating systems. Check out http://www.mysql.com/downloads.

This section describes the installation of the binary version of MySQL under Windows and under Linux.

Remark

When one speaks of MySQL one is usually referring to the database server. A MySQL installation thus usually means the installation of the server, where usually all the client tools are installed at the same time, so that the server can be administered. This is the subject of the current section.

If MySQL is already installed on another computer and you wish only to access that installation, then you need only the client tools. The next section discusses such aseparate installation.

Tip

If you are developing a web application that is to be installed by an external Internet service provider, you should find out what version is being used. Many ISPs are reluctant (with good reason) to install the most recent version.

If your ISP is using an older version of MySQL (for example, 3.23.n), then it is often a good idea to use the older version for your own development. In this way you avoid the situation in which you have used certain functions that will be unavailable on the system of your ISP.

Installation Under Windows

MySQL for Windows is available as a WinZip archive. To install it, copy the contents of the archive to a temporary directory and there launch setup.exe. (With current versions of WinZip you can simply launch setup.exe. WinZip extracts the necessary files on its own into a directory and then deletes the archive when the installation is complete.)

The MySQL setup program assumes that under Windows, the Microsoft installer (MSI) is available. In current versions of Windows that is automatically the case. If you are using an older version of Windows, you can obtain MSI without charge at the following address:

http://support.microsoft.com/default.aspx?kbid=292539

The setup program itself offers no surprises. You may specify the installation location and select the components to be installed. If you stick with the default settings, then the database system, the documentation, and a default setting for access rights (the privileges database mysql) are installed into the directory C:\mysql. The space requirement is about 50 megabytes (where after the first launch an additional 20 megabytes for database files are needed). Optionally, you can also install files for benchmark tests as well as libraries and include files (if you wish to develop MySQL clients with C/C++).

Warning

The setup program takes no notice of any existing MySQL installations. If you wish to update an installation, you should first secure the existing databases, then deinstall the old version of MySQL, and then install the new version. This process is described more fully later.

MySQL can be run as a freestanding program or as a service. The latter variant, however, can be used only under Windows NT/2000/XP.

Remark

In this book it is generally assumed in descriptions of Windows NT/2000 that the installation of programs is executed by the administrator (or by a user with equivalent privileges) and that programs such as WinMySQLadmin are also executed by such a user.

An installation without administrator privileges is usually impossible. Even with the use of programs without administrator privileges there can arise problems with access privileges, related to the manner in which access rights to files and processes are managed under Windows NT/2000. (A description of the access system of Windows NT/2000 is beyond the scope of this book.)

Initial Configuration and Launch

For the initial launch of a MySQL server your best bet is to use the administration program mysql\bin\winmysqladmin.exe included with MySQL. (Simply launch the program in Windows Explorer with a double click.)

Tip

WinMySQLadmin was originally conceived as the user interface to MySQL. The program was never able to fulfill this task adequately. In the meantime, there have appeared many other programs that are better suited for this. Therefore, the further development of WinMySQLadmin was stopped. Nevertheless, WinMySQLadmin has a reason for being: The program is extremely practical for carrying out certain elementary administrative tasks:

  • starting and stopping the MySQL server

  • setting up the MySQL server (only under Windows NT/2000/XP)

  • setting up and editing a configuration file for the MySQL server

On initial launch of WinMySQLadmin, the program asks for a user name and password. The program will use this information in the future in establishing a connection to the MySQL server. Since WinMySQLadmin, as mentioned, is seldom used as an administration program, an administration login is not required. Therefore, you should input any character string that you like in the dialog; WinMySQLadmin rewards this input by establishing the MySQL configuration file Windows\My.ini, which contains, in addition to the login information, other important information for launching MySQL.

Warning

Note that the user name and password are stored in plain text in the file Windows\my.ini. Therefore, be sure not to provide a password that might endanger the security of the system.

The following lines of code show the MySQL configuration file windows\my.ini generated by WinMySQLadmin. Of greatest importance are the settings basedir and datadir. They specify the MySQL installation directory and the directory with the MySQL databases. This information is required if MySQL was installed in a directory other than c:\mysql:

# MySQL-configuration file Windows\My.ini
[mysqld]
basedir=C:/mysql410
datadir=C:/mysql410/data
[WinMySQLadmin]
Server=C:/mysql410/bin/mysqld-nt.exe
user=abc
password=efg

After the configuration file has been set, WinMySQLadmin establishes the MySQL server as a Windows service (Windows NT/2000/XP only) and starts the service (see Figure 2-1).

Click To expand
Figure 2-1: WinMySQLadmin

Then WinMySQLadmin establishes a start link in the Autostart directory so that the program will be launched automatically in the future. That is important especially under Windows 9x/ME: There the MySQL server cannot be started as a Windows service and therefore must be launched by WinMySQLadmin.

WinMySQLadmin minimizes itself directly after launch into a traffic-light-shaped icon in the task bar. The light gives the status of the MySQL server. (Green means that the server is running.) Using the light's context menu you can start or stop the server or open WinMySQLadmin's window.

If there are problems at launch, then launch the user interface of WinMySQLadmin with the right mouse button using SHOW ME. The dialog form START CHECK provides information about possible causes of the launch failure. In the dialog MY.INI SETUP you can examine the file Windows\My.ini, make changes, and then save via SAVE. HIDE ME in the dialog sheet ENVIRONMENT minimizes WinMySQLadmin again into the traffic-light icon.

Starting and Stopping MySQL service under Windows NT/2000/XP

Once MySQL has been set up as a service, it will be started automatically each time the computer is booted up. If you wish to start or stop MySQL manually, you can use the Windows service manager instead of WinMySQLadmin.exe. To do this, execute SETTINGS |CONTROL PANEL |ADMINISTRATIVE TOOLS |SERVICES, select the service mysql, and then click on the desired button (START, STOP, etc.); see Figure 2-2.

Click To expand
Figure 2-2: Starting and Stopping MySQL as a service
Tip

You are not dependent on WinMySQLadmin to set up your MySQL server as a service. If you want to do this task manually, then launch mysql\bin\mysqld-nt.exe in an input command window with the option --install. To deinstall the service, use the option --remove.
Pointer

The configuration file windows\My.ini of course offers countless other configuration possibilities. Information on this can be found in Chapter 19. But first, it is important to get the MySQL server up and running.

MySQL Server Variants Under Windows

Under Windows there is a selection of several variants of the MySQL server: mysqld-nt.exe, mysqld-max.exe, mysqld.exe, etc. The following table shows the differences among them:

Version | Debugging information | Additional functions (BDB tables, SSL, etc.) | Optimized for Windows
mysqld.exe | Yes | Yes | No
mysqld-opt.exe | No | No | No
mysqld-nt.exe | No | No | Yes
mysqld-max.exe | No | Yes | No
mysqld-max-nt.exe | No | Yes | Yes

The debugging information is relevant only if the MySQL server crashes (which has never happened to me with stable MySQL versions) and you wish to send the debugging information to the developer or use the debugger yourself to search for errors.

Additional functions are those that are rarely used or not yet mature. Currently (MySQL 4.0.12), this includes BDB tables and SSL encryption.

"Optimized for Windows NT/2000/XP" means that the server can be executed as a Windows service and that named pipes are available (these represent a seldom used form of communication between the server and Windows clients that run on the same computer).

Normally, you should use mysqld-opt.exe under Windows 9x/ME, and mysqld-nt.exe under Windows NT/2000/XP. The selection of the version of mysqld is most simply accomplished in WinMySQLadmin in the dialog sheet MY.INI SETUP.

Manually Installing and Deinstalling MySQL as a Windows Service

As we have described in some detail, the program WinMySQLadmin offers the simplest way of installing the MySQL server under Windows NT/2000/XP as a Windows service.

If the day were to come when this program was no longer provided with MySQL, then there would presumably be an improved setup program. In any case, it is good to know that you can easily install and deinstall Windows services manually.

Installation is accomplished by first opening an input command window and executing the following commands:

>cd mysql\bin
> mysqld-nt --install

With this, the program is installed as a Windows service with the name mysql, but not launched. In the future, the service mysql will be automatically launched at computer bootup, and terminated when the computer is shut down.

There are several installation variants:

  • Instead of mysqld-nt, you can use one of the other mysqld variants mentioned earlier.

  • If you do not want the service to start up automatically at system startup, then instead of --install, use the option --install-manual.

  • If do not want to install the service under the name mysql, then with the --install option provide another name (--install name). This makes sense only in special cases, for example, with a parallel installation of several MySQL versions.

To start or stop the service manually, execute one of the following two commands:

> net start mysql
> net stop mysql

To stop the server, you can alternatively execute mysqladmin -u root -p shutdown, where you will have to provide the root password for the server.

To deinstall the service, use the option --remove:

> mysqld-nt --remove
Tip

If you wish to start the MySQL server manually under Windows NT/2000/XP as a standalone program (not as a service), then execute the command mysqld --standalone or mysqld --console. In both cases the input window is blocked until the server is ended with mysqladmin shutdown. The difference between --console and --standalone is that in the former, the status information is displayed directly in the input window.

Installation Under Linux

MySQL is included in most Linux distributions. The installation is here confined to installing the MySQL program package with the distribution's package manager. This is described below in detail for Red Hat and SuSE Linux.

Red Hat: Installation of the Included Version

Before MySQL is installed, some Perl packages must be installed, if they have not been installed already. All the packages are to be found on CD 2. The option --nodeps is necessary in the second rpm command, since on account of an error in Red Hat 9, the packages perl-DBD-MySQL and mysql are mutually dependent in that each package should be installed only after the other—a classic "Catch 22." The meaning of --nodeps is that package dependencies are to be ignored, which overcomes the dependency problem:

root# rpm -i /mnt/cdrom/RedHat/RPMS/perl-DBI-n.i386.rpm
root# rpm -i --nodeps /mnt/cdrom/RedHat/RPMS/perl-DBD-MySQL-n.i386.rpm
root# rpm -i /mnt/cdrom/RedHat/RPMS/perl-CGI-n.i386.rpm

MySQL 3.23.54 is included with Red Hat 9. MySQL is divided into four packages, whose RPM files are located on CD 2:

  • mysql-version.rpm contains the client programs, including the shared libraries, which are used by many programs to access MySQL. The client programs (mysql, mysqladmin, etc.) aid in the administration of MySQL.

  • mysql-server-version.rpm contains the actual MySQL server.

  • mysql-devel-version.rpm contains the libraries and include files for program development in C.

Usually, only the first two of these packages need to be installed.

root# rpm -i /mnt/cdrom/RedHat/RPMS/mysql-n.i386.rpm
root# rpm -i /mnt/cdrom/RedHat/RPMS/mysql-server-n.i386.rpm

To determine whether MySQL is already installed on your computer (and if so, in which version), execute the following commands:

root# rpm -qa | grep -i
mysql mysql-3.23.54a-10
mysql-server-3.23.54a-10
perl-DBD-MySQL-2.1021-3

To try out your newly installed MySQL server, execute the following command:

root# /etc/init.d/mysqld start

If you want the MySQL server always to start automatically, you must add mysqld to the appropriate init-V run levels. The simplest way of proceeding is to make this setting with chkconfig:

root# chkconfig --level 235 mysqld on

As database directory, /var/lib/mysql is used.

SuSE: Installation of the Included Version

SuSE 8.2 is included with MySQL 3.23.55. To install it, launch the YaST2 module INSTALL SOFTWARE, choose the filter SEARCH, and input mysql. Included in the search result will be the following packages:

  • mysql contains the actual MySQL server.

  • mysqlcc contains the MySQL Control Center (a graphical user interface for administration).

  • mysql-client contains the client programs (mysql, mysqladmin, etc.) for administration of MySQL.

  • mysql-shared contains the shared libraries, which are used by many programs to access MySQL.

  • mysql-devel contains libraries and include files for program development in C.

  • mysql-max contains the Max version of the MySQL server, which, among other things, supports BDB tables.

As a rule, it suffices to install the three packages mysql, mysql-client, and mysql-shared. If you want to try out the MySQL Control Center, then install mysqlcc as well. The other packages are not of immediate interest. MySQL can be launched manually after installation with the following command:

root# /etc/init.d/mysql start

To have MySQL launch automatically at startup, execute the following command:

root# insserv mysql

As database directory, /var/lib/mysql is used.

Installation of the MySQL Linux Package of mysql.com

Every Linux distribution comes with its own MySQL packages. Their installation proceeds without difficulty (see the previous section), but the following two issues should be taken into consideration:

  • The included versions are often quite old. Update packages are produced generally only when serious security problems need to be fixed.

  • In the past, the versions provided were generally unstable. For example, with both Red Hat 8 and SuSE 8.1 the included MySQL server would crash as soon as a MySQL client from another server attempted to make itself known. (The server is restarted automatically; the client receives an error message: Lost connection to MySQL server during query.) Both companies quickly provided updates to the server and affected libraries, but the problem caused a great deal of confusion and anger among MySQL beginners. (The versions provided with RedHat 9 and SuSE 8.2 appear to be more stable.)

On the web site mysql.com it is therefore recommended that one use only MySQL packages from that site. It is with these packages only that one can be sure that they will be optimally compiled.

Warning

Alas, following this suggestion is problematic. Every Linux distributor has its own ideas of how the software packages and their dependencies should be managed, in which directories the files should be installed, how the programs should be executed on system startup, etc.

For this reason, only experienced Linux users should install packages that are not explicitly prepared for the Linux distribution. It would also be good to have some MySQL experience in case the installation runs into difficulties.

If the installation of new packages leads to a major change in the MySQL version (such as MySQL 3.23.n to 4.0.n, or from 4.0.n to 4.1.n), and you wish to transfer existing data, then you should read the sections on backup and data migration in Chapter 10.

If you have not been frightened off by this warning and wish to try out the latest version of MySQL, then here are a few tips on installation. We begin with the assumption that you already have completed an installation of MySQL, Apache, PHP, Perl, etc., based on the packages included in the distribution. The goal now is to deinstall the existing version of MySQL and replace it with a more current version.

Organization of the mysql.com Packages (RPMs)

You will find at the download site mysql.com the following packages for the current version of MySQL in RPM format for Intel-compatible processors:

mysql-server-n.rpm | contains the MySQL server
mysql-client-n.rpm | contains client programs (mysql, mysqladmin, etc.)
mysql-shared-n.rpm | contains the shared libraries used by many programs to access MySQL
mysql-devel-n.rpm | contains libraries and include files for program development in C
mysql-embedded-n.rpm | contains a special version of the MySQL server that can be integrated into other programs (for C developers only)
mysql-bench-n.rpm | contains scripts with benchmark tests
mysql-Max-n.rpm | contains the Max version of the MySQL server, which supports BDB tables

Installation of mysql.com RPMs Under Red Hat 9

The following instructions are for MySQL 4.0.12 under Red Hat 9. With other versions of MySQL and Red Hat there might be variations. I have tested the installation as thoroughly as I could, but there is always the possibility that some problems could arise:

  • Stop the server (/etc/init.d/mysqld stop).

  • If you have changed the MySQL access privileges, then create a backup copy of the directory /var/lib/mysql/mysql with the database mysql.

  • Create a backup of /etc/my.cnf.

  • Create a backup copy of the library libmysqlclient.so.10.(This is a library that dynamically links MySQL programs used to access the MySQL server. Under RedHat 9 this affects, for example, PHP and Perl.)

    root# cd /usr/lib
root# cp -a mysql/ mysqlbak

  • Deinstall the following packages (those that are installed): mysql, mysql-server, and mysql-devel:

    root# rpm -e mysql-server
root# rpm -e --nodeps mysql

  • Install the RPM package of mysql.com with the following commands:

    root# rpm -i MySQL-server-n.rpm
root# rpm -i MySQL-client-n.rpm
root# rpm -i MySQL-shared-n.rpm

    The newly installed server is launched at once. No configuration file /etc/my.cnf is installed (at least not with the version 4.0.12 tested). As before, /var/lib/mysql is used as data directory.

    Make sure that the MySQL server will be launched automatically by each system startup:

    root# chkconfig --level 235 mysql on

    Note that the script for starting and stopping the server is now /etc/init.d/mysql (and no longer mysqld).

  • Restore the libmyclient.so.10 libraries:

    root# cd /usr/lib
root# rm -rf mysql
root# mv mysql-bak mysql
root# ldconfig

    There are now two versions of libmyclient installed: libmyclient.so.10 in the directory /usr/lib/mysql, and libmyclient.so.12 in the directory /usr/lib.

    The older version is required for various Red Hat original packages (Perl, PHP, etc.), the new versions for MySQL client programs that use the new functions of the current version of MySQL.

  • If necessary, restore the mysql database and /etc/my.cnf. After a significant version change you will have to execute the script mysql_fix_privilege_tables (see Chapter 10).

Installation of mysql.com RPMs Under SuSE 8.2

The following instructions are for the installation of MySQL 4.0.12 under SuSE 8.2. These can differ for other versions of MySQL and SuSE. I have tested the installation, but it is still possible for problems to arise.

  • Stop the server (/etc/init.d/mysqld stop).

  • If you have changed the MySQL access privileges, then create a backup copy of the directory /var/lib/mysql/mysql with the database mysql.

  • Create a backup copy of /etc/my.cnf.

  • Start YaST2, select the filter SEARCH, and search for currently installed mysql packages. Select the following packages for deinstallation (those that are installed): mysql, mysql-bench, mysql-client, mysql-devel, and mysql-max. Note that you may not deinstall mysql-shared!

    Click on the button CHECK DEPENDENCIES. A dialog appears in which YaST2 dependency conflicts are determined (that is, that other installed packages may no longer function if the MySQL packages are deinstalled). For solving conflicts choose the option IGNORE CONFLICT AND RISK SYSTEM INCONSISTENCIES.

    Deinstall the selected packages.

  • Install the RPM packages of mysql.com with the following commands:

    root# rpm -i MySQL-server-n.rpm
root# rpm -i MySQL-client-n.rpm
root# rpm -i MySQL-shared-n.rpm

    The newly installed server is launched at once. No configuration file /etc/my.cnf is installed (at least not with the version 4.0.12 tested). As before, /var/lib/mysql is used as data directory.

    Note that now two MySQL shared packages are installed simultaneously. One of them comes from SuSE and makes available the library libmysqlclient.so.10, which is needed by Perl or mysqlcc; the other package comes from msql.com and contains the new version ysqlclient.so.12 for programs that wish to use the new MySQL functions.

  • If necessary, restore the mysql database and /etc/my.cnf. After a major version change you must also execute the script mysql_fix_privilege_tables (see Chapter 10).

Incompatibilities After Installation

Red Hat: The Init-V script for starting the MySQL server now has the name mysql (instead of mysqld).

Both Distributions: A further difference affects the internal management of the package dependencies. If after the MySQL update you install other packages that require MySQL, it can happen that the distributions do not recognize that MySQL has already been installed. With Red Hat, in the installation of such packages you should use rpm with the option --nodeps (not redhat-config-packages). With SuSE you can specify in YaST2 that the package dependencies are to be ignored.

Finally, problems are also to expected in distribution updates. In an update to a new version of the Linux distribution, MySQL is probably left untouched and will not be updated automatically.

Testing the Installation

To test whether the MySQL server runs and whether you can use it, launch the program mysql. Under Linux you simply execute mysql in a shell window. Under Windows launch mysql/bin/mysql.exe with a double click in Explorer. In this program you execute the command status. Under Unix/Linux the result should look like what is pictured in Figure 2-3.

Click To expand
Figure 2-3: Testing the MySQL installation under Linux

If that doesn't work, check whether the MySQL server is running at all. Under Linux you would execute ps -ax. It is normal that mysqld appears several times in the resulting list. (The server was not started several times, but divided itself into several threads.)

root# ps -ax | grep mysqld 2725 pts/2
2725 pts/2    S    0:00 /bin/sh /usr/bin/safe_mysqld . . .
2756 pts/2    S    0:00 /usr/sbin/mysqld . . .
2757 pts/2    S    0:00 /usr/sbin/mysqld . .
2758 pts/2    S    0:00 /usr/sbin/mysqld . . .
2799 pts/2    S    0:00 grep mysqld

Under Windows NT/2000, take a look at the task manager. There should be a process running called mysqld*.exe.

If the MySQL server is not running, you will probably have to start it manually. Under Linux you would use the init-V script mysql[d] start. Under Windows use instead the program WinMySQLadmin.

Remark

The name mysqld is short for the MySQL daemon. This is simply another name for the MySQL server.

Under Unix/Linux it is usual to call auxiliary programs running in the background "daemons." (In Windows NT/2000 such programs are called "services.")

Security

After a default installation, MySQL is insecure. Anyone can enter without a password as root and has unlimited privileges. (Under Windows the default installation is even more insecure, if that is possible.)

As long as you are not storing any critical data in MySQL, you can continue to use the database system unsecured (just be aware of the security risks, however). If you value security, then you will find in Chapter 9 an extensive description of the MySQL access system, including a step-by-step introduction to beginning to secure your system.

Updating MySQL

You will find the most recent versions of MySQL in various formats (ZIP, RPM) at www.mysql.com.

Caution

In general, it is recommended that you back up all MySQL databases as well as the configuration files Window\my.ini and etc/my.cnf before undertaking a MySQL update.

If you update MySQL within a main version, that is, a small version change such as 4.0.12 to 4.0.13, then you can use your database files without change. The backup is merely a security measure.

In the case of a large version change (such as from 4.0.n to 4.1.n), it could happen that you will have to reconfigure the databases. (In the past, MySQL was always backward compatible, but that is no guarantee for the future.) Further information on making a backup and on migrating between databases of incompatible versions of MySQL, as well as general information on mysqldump, can be found in Chapter 10. This section assumes that a small version change is carried out and that therefore the database files can be used without alteration.

Updating Under Windows

  • Stop the server (for example, with WinMySQLadmin).

  • By all means, execute a backup of the database mysql. This database is deleted by deinstallation of MySQL. It contains the settings of all access privileges. The simplest thing to do is to copy the directory Mysql/data/mysql to another location.

  • Install the new version of MySQL in the same directory as before.

  • Restore the backup copy of the mysql database. Also, copy the backed-up directory mysql into the MySQL data directory Mysql/data.

  • Start the server.

Updating Under Linux

  • Stop the server ((/etc/init.d/mysql[d]).

  • Execute a backup of the database mysql and the file /etc/my.cnf.

  • Deinstall the MySQL packages with rpm -e or another package management tool (SuSE: yast2).

  • Install the updated MySQL packages.

  • Restore the mysql database. (If the MySQL server was automatically launched after installation, you must first terminate it.)

  • Launch the MySQL server.


MySQL (Client Installation)

All of the client programs are automatically installed with the MySQL installation described above. Often, however, the server runs on a different computer, and you require only the client tools to access the external server. In this case, a space-saving client installation is sufficient.

With a client installation, only various administration programs (mysql, mysqladmin, mysqldump, etc.) as well as the related libraries and character set files are installed. Then you can use these programs to create new databases on the external MySQL server, read existing data, execute SQL commands, and so on.

Remark

In some situations, you can dispense entirely with the installation of client tools on the local computer, namely, when an administration tool is installed on the MySQL computer that runs locally (for example, phpMyAdmin) and can be used over the Internet. If the MySQL server runs on the computer of an ISP, then that is the usual situation; see the last section of this chapter.

Windows

The zip archive Winclients-version.zip with all the files for a client installation can be found at the following address:

http://www.mysql.com/downloads/os-win32.html

Unfortunately, the client installation files were completely outdated as this book was being written. For this reason, it is better to execute a complete server installation, even if the server itself is not even going to be used.

Unix/Linux

The most current Linux distributions provide packages for client installation. Simply install the packages mysql-n.rpm and mysql-shared-n.rpm (not mysql-server-version.rpm). You can find such RPM files for the client installation at the download site www.mysql.com, and, in contrast to the comparable Windows package, always in the latest version.


Apache

Pointer

This section describes a basic installation of Apache and some elementary configuration steps. Detailed information on the countless configuration possibilities as well as security of the installation can be found in books on Apache and at the Apache documentation site:

http://www.apache.org/docs/

Apache 1.3 versus Apache 2.0

At present, there are two versions of Apache available: Apache 1.3.n and Apache 2.0.n. Both versions are considered basically stable. However, in comparison to Apache 1.3, version 2.0 offers extensive new developments. Many supplementary modules to Apache 2.0, however, are not as fully mature and well tested as those of version 1.3. This applies particularly to the frequently installed PHP module. (The combination Apache 2.n and PHP 4.3 was still in an experimental stage as the work on this book was being completed.)

For this reason, you will still see Apache 1.3 on many Internet servers (and thus with many Internet service providers) for a while to come. For the examples of this book it does not make a great deal of difference whether you use Apache 1.3 or 2.0. I have tested the PHP and Perl–CGI examples under both versions without encountering any difficulties. There are few differences between the two versions in their basic configurations.

If you are working under Linux, then you should install the default version of Apache that comes with your distribution. If you set up a test system under Windows, then the choice is yours. This book describes the installation of version 2.0. If you prefer to install version 1.3, you will encounter a few differences from what is presented here.

Apache 2: Installation Under Windows

Deinstalling IIS

If youare working with a Windows Server version, then you should first make sure that the Microsoft Internet Information Server (IIS for short) is not running. The standard installation of Apache (port 80) does not work if IIS is running in parallel (for which there is seldom a reason in any case).

For checking or deinstallation of IIS execute SETTINGS |CONTROL PANEL |ADD/REMOVE PROGRAMS |ADD/REMOVE WINDOWS COMPONENTS. If necessary, deactivate the option IIS in WINDOWS COMPONENTS WIZARD, upon which the program will be terminated and deinstalled.

Installing Apache

Apache is available at www.apache.org and at various mirror sites as an installation program for Windows. Download the file apache_2.n_xxx.msi to your computer and launch the installation program in Windows Explorer with a double click. A dialog box appears for the basic configuration (see Figure 2-4). You will have to give the domain name (e.g., sol) and server name (e.g., uranus.sol). This information is needed even if your computer runs as a standalone test computer. As a rule, give simply the domain or workgroup name as well as the computer name. If you do not know what these names are on your computer, you can find them in the dialog SETTINGS |CONTROL PANEL |SYSTEM |NETWORK IDENTIFICATION.

Then you should specify the e-mail address of the administrator of the web server (you!) and whether the web server should use port 80.

After installation, Apache will be set up automatically as a service and launched immediately. In the right part of the task bar a small icon will appear that shows the state of Apache (a green arrow if the program is running). This icon can be used for starting and stopping Apache.

Warning

Note that your web server is now accessible from the Internet if you are connected to it. This holds even if you are connected through a dial-up modem. The web server is not reachable through an easily guessed name like http://www.kofler.cc, but via the current IP number given by your Internet service provider (e.g., http://172.12.34.45). You should make sure that no confidential information can leak out of your system, even in a test installation. The best security is a firewall that generally keeps out access from the outside.
Click To expand
Figure 2-4: Apache basic configuration

Installation Under Linux

Apache is normally automatically installed under Linux. If that is not the case in your situation, you will have to install the relevant package (usually apache-version.rpm).

Red Hat

With Red Hat use the installation redhat-config-packages and choose the group Webserver.In addition to Apache 2, various associated programs are also installed.

SuSE

With SuSE launch the YaST2 module INSTALL SOFTWARE and install either the group SIMPLE WEBSERVER (Aache 1.31) or SIMPLE WEBSERVER WITH APACHE 2. Some additional packages will also be installed. The following description pertains to Apache 2.

Configuration and Start

Configuration File

Almost all Apache settings are managed via the configuration file httpd.conf. Important for the first startup is generally only the setting for ServerName. You should provide the network name of your computer, for example, the following:

# change in httpd.conf
ServerName saturn.sol # Here give the name of your computer

If the ServerName installation fails, Apache attempts to discover the computer name on its own. This works sometimes, but not always. Where httpd.conf is located depends on the operating system:

Windows | Programs\Apache Group\Apache[2]\conf\httpd.conf
Red Hat Linux | /etc/httpd/conf/httpd.conf
SuSE Linux | /etc/apache2/httpd.conf

Launching Apache

To launch the program after setting httpd.conf, the following commands are necessary:

Windows NT/2000/XP | Apache is executed by default as a service. You can check this in the dialog SETTINGS | CONTROL PANEL | ADMINISTRATIVE TOOLS | SERVICES.
Red Hat Linux | root# /etc/init.d/httpd start
SuSE Linux | root# /etc/init.d/apache2 start

To have Apache executed automatically under Linux at computer startup, execute the following commands:

Red Hat Linux | root# chkconfig --level 35 httpd on
SuSE Linux | root# inssrv apache2

Testing the Apache Installation

You can check with your web browser that everything is functioning correctly by calling up the page http://localhost or (following the example of this chapter) http://saturn.sol. This address leads to the start page of Apache, where temporarily only a single test page is displayed (see Figure 2-5).

Click To expand
Figure 2-5: Apache's default start page
Remark

The name localhost is valid both under Windows and under Unix/Linux as the network name for the local computer. The address http://localhost therefore corresponds to http://your.computer.name (or whatever you have named your computer).
Restarting Apache After Changing httpd.conf

After a change in httpd.conf, you must restart Apache so that the new configuration becomes effective. Depending on the operating system, the restart is carried out in one of several ways:

Windows | First, execute START |PROGRAMS |APACHE |STOP APACHE;then START APACHE.You can also restart Apache 2 in the Apache Monitor (visible as an icon in the right part of the task bar). Under Windows NT/2000/XP there is a third variant: In SET- TINGS |CONTROL PANEL |ADMINISTRATIVE TOOLS |SERVICES select the service Apache and click first on STOP in the SERVICES dialog, and then on START.
Red Hat Linux | root# /etc/init.d/httpd restart
SuSE Linux | root# /etc/init.d/apache2 restart
Tip

If you encounter a (syntax) error in changing httpd.conf, the next Apache restart will fail. Information on the cause of the error can be found under Windows with START |PROGRAMS |APACHE |CONFIGURE APACHE |TEST CONFIGURATION.
Creating an Alias (Virtual Directory)

In the default installation, Apache enables access to all files and subdirectories of a root directory. The path to this directory varies according to the operating system (in fact, according to the setting of DocumentRoot in httpd.conf). The following list shows the usual default settings:

Windows | Programs\Apache Group\Apache[2]\htdocs
Red Hat Linux | /var/www/html
SuSE Linux | /srv/www/htdocs

With an alias it is possible to associate a virtual directory on the server with an arbitrary directory on the hard drive on the server. For example, via Apache, with the path http://computer.name/virtual you can access the actual directory e:\data\real or /usr/share/help.An alias is also helpful in simplifying access over the web server to directories located at various places on the hard drive.

To try out the PHP examples in this book you will not need an alias. You can simply copy the examples to the DocumentRoot directory. But if you would rather store the examples at a different location on your hard drive, you will have to add an alias line in httpd.conf according to the following template:

#in httpd.conf
Alias /mytest/ "E:/directory/subdirectory/"

In the definition of alias directories the following rules should be followed:

  • All alias definitions usually occur in httpd.conf in a block that is introduced by <IfModule mod_alias.c> and ended with </IfModule>. In the Apache 2 default installation these tags are lacking, and the correct position can be recognized by the comment #Aliases: Add here as many aliases as you need ... .

  • Under Windows the directories must be indicated with a forward, not a backward, slash.

  • Directories (e.g., e:/data/real/) must end with a forward slash.

  • After the alias definition, the properties of the directory must be set with <Directory "directory">.A <Directory> block might look as follows:

    <Directory "E:/dir/subdir/">
   Options Indexes MultiViews
   AllowOverride All
   Order allow,deny
   Allow from all
</Directory>

    This means that when the directory does not contain the file index.html (option Indexes), a list of all files in this directory will be displayed; that the correct version of the files will automatically be used if there is a choice from among several languages (MultiViews); that .htaccess will be taken into account and all relevant options will be processed (AllowOverride All); and that everybody can access the directory (Order and Allow).

The option Indexes is useful for testing, but it should be eliminated in production, because it represents a security risk. (It is possible that a directory contains files that should not be visible to all. Indexes reveals the names of these files.)

For the change in httpd.conf to become effective, Apache must be restarted.

Access Protection for Individual Directories (.htaccess)

There are situations in which not all of the directories managed by Apache should be universally accessible. For example, if you install phpMyAdmin for administration of MySQL at an Internet service provider, these administrative tasks should be available to you alone, and not to every web surfer who happens by.

Password File

The simplest method of protecting individual directories is offered by the file .htaccess, which is placed in the directory in question. However, it is first necessary to set up a password file, which in the following example is named site.pwd. This file can, in principle, be located just about anywhere, though for reasons of security it is recommended to select a directory that is not accessible from without (that is, outside of the directories reachable by Apache).

The password file is generated with the Apache auxiliary program htpasswd. (If you are working under SuSE with Apache 2, the command is htpasswd2.) Choose the option -c (create), and specify the name of the password file and a user name. The program will ask you for the desired password, and then it generates the password file, in which the user name appears in plain text, the password in encrypted form.

> htpasswd -c site.pwd myname
New password: **********
Re-type new password: **********
Adding password for user myname

To add an additional combination of user name and password to a preexisting password file, execute htpasswd again, this time without the option -c:

> htpasswd site.pwd name2
New password: **********
Re-type new password: **********
Adding password for user name2

The resulting password file will look something like this:

myname:$apr1$gp1 ... ..$qljDszVJOSCS.oBoPJItS/
name2:$apr1$A22 ... ..$OV01Nc1FcXgNsruT9c6Iq1
Remark

If you wish to execute htpasswd.exe under Windows, you have to open a command window (command prompt) and specify the complete path name inside quotation marks (usually "Programs\Apache Group\Apache\bin\htpasswd.exe").

If you want to create or enlarge the password file in a directory of an Internet service provider, then use telnet or ssh to access the computer of the ISP (in order to execute htpasswd there). Many ISPs make available other configuration aids.
Warning

.htaccess files are effective only when allowed in the Apache configuration file httpd.conf.Sometimes, the Apache default configuration contains the entry AllowOverride None,andthen the .htaccess files remain useless.

For .htaccess files to be considered, the relevant directory (within a <Directory> group) must have specified the option AllowOverride All or AllowOverride with a combination of AuthConfig, FileInfo, Limit,and Options.

Remark

Under Windows, it is impossible with Windows Explorer to create a file with the name .htaccess. Explorer believes that this is an invalid name. Normally, this is no problem, since you can create .htaccess with the command htaccess. If for some reason you need to use Explorer, then first, name the file htaccess.txt and then, use an input window (DOS window) and rename the file with RENAME htaccess.txt .htaccess

Building the .htaccess File

An .htaccess file protects the entire contents of a directory (including all subdirectories). The file normally consists of the following four lines. Here AuthUserFile specifies the complete file name of the password file:

AuthType Basic
AuthUserFile /www/username/htdocs/_private/site.pwd
AuthName "myrealm"
Require valid-user

AuthName denotes the realm within which access is valid. The point is that you do not have to execute a login every time want to access various directories protected by .htaccess. Once you have logged in with a particular AuthName, this login remains valid for all further directories with this AuthName indicator.

For .htaccess this means that you specify the same AuthName character string for directories with a common login. Conversely, for varying directories you may specify varying domain names, in which case a new login is required for each directory.

Require valid-user means that every valid combination of user name and password is allowed. Alternatively, you can specify here that a login is permitted only for particular users:

Require user myname name2 name3

Access to a Protected Directory

As soon as you attempt to read a file from a protected directory with a browser, the webbrowser presents a login dialog (see Figure 2-6).

Pointer

Further details on user authentication and .htaccess configuration can be found at the following web addresses:

http://apache-server.com/tutorials/ATusing-htaccess.html
http://www.apacheweek.com/features/userauth
Click To expand
Figure 2-6: Access to a web directory protected by .htaccess


PHP

PHP stands for PHP Hypertext Preprocessor (which is to say that we are dealing with a recursive abbreviation, so beloved in the Unix community). PHP enables the insertion of script code into HTML files. It works like this: Every time a surfer requests a file with the identifier *.php, the web server (Apache, for example) transmits this file to the PHP interpreter. This, in turn, evaluates the PHP code and constructs the resulting HTML document, which is then returned to the web server. The web server then directs the file to the surfer. PHP thus offers a simple way of generating dynamic web pages. No assumptions need be made about the client.

The examples in this book were tested with the following versions of Apache and PHP:

  • Windows 2000: Apache 1.3.n and Apache 2.n with PHP 4.3

  • RedHat 8.0: Apache 2.n with PHP 4.2

  • RedHat 9.0 beta 3: Apache 2.n with PHP 4.2

  • SuSE 8.1: Apache 1.3.n with PHP 4.2

  • SuSE 8.2 beta 5: Apache 2.n with PHP 4.3

Remark

I would like to mention once again that as of March 2003 the combination of Apache 2 and PHP 4.2/4.3 still has the status experimental and is not to be recommended for use in production. No problems arose in my tests, but for web servers in production use the present recommendation is Apache 1.3.n.

Perhaps the Apache/PHP combination will have matured by the time you are reading this book. With regard to the configuration possibilities described in this book, nothing changes in moving from Apache 1.3 to 2.0.

The CGI Version Versus the Apache Module

There are two basic ways that Apache can run the PHP interpreter: as a freestanding program and as a module.

  • In the first variant, the PHP interpreter is executed as a CGI program. The program must be restarted for the translation of each *.php file, which is relatively inefficient. The advantage of the CGI variant is that installation and configuration are simpler.

  • In the second variant, the PHP interpreter is run as an Apache module. The module is a sort of extension to the Apache program. It is loaded once and then remains in memory. The advantage of the module variant is that execution is considerably more efficient. Furthermore, some additional functions are available under PHP (for example, HTTP authentication functions).

    Under Windows, Apache has been extended to include the PHP module, using SAPI (Server Application Interface).

    Under Unix/Linux, on the other hand, the PHP module is usually included in the form of a Dynamic Shared Object (DSO). Both interfaces enable the subsequent addition or exchange of Apache modules without Apache itself having to be recompiled.

    Remark

    You can easily configure Apache on your test system in such a way that it can deal with *.php, *.php3, and *.php4 files. However, on your ISP's system, that is not the case; that is, your ISP generally prescribes a particular file identifier (usually *.php). To avoid problems later with the installation of your PHP files on the ISP computer, you should use the same file identifier in your test environment as that of your ISP.

    In this book, we generally use the file identifier *.php. If your web server is configured in such a way that the file identifier *.php3 or *.php4 is prescribed, then you must make the appropriate changes. Furthermore, you must adapt all the links within HTML and PHP code (this affects particularly <a href= ... > and <form action= ... >).

PHP4 Installation Under Windows

At the web site http://www.php.net/ you will find the most recent version of PHP in two variants: as a Zip archive (which can be opened with the free program WinZip) and as an installation program. Since the installation program is dependent on the Internet information server, while in this book Apache is used as the web server, you should choose the Zip variant.

Installation comprises the following steps:

  • Simply copy all the files in the Zip archive into an arbitrary directory. (The following examples assume that the installation directory is C:\php-430.)

  • Copy the file C:\php-430\php4ts.dll into the directory sapi in your PHP installation (C:\php-430\sapi).

    Insert the boldface lines below into the Apache configuration file httpd.conf.

    #in httpd.conf for Apache 2
...
LoadModule php4_module c:/php-430/sapi/php4apache2.dll
AddType application/x-httpd-php .php .php3 .php4
...

    If youare using Apache 1.3, the configuration looks like this:

    #in httpd.conf for Apache 1.3
...
LoadModule php4_module c:/php-430/sapi/php4apache.dll
AddModule mod_php4.c
AddType application/x-httpd-php .php .php3 .php4
...

  • Restart Apache. If a warning is displayed with Apache 1.3 that the mod_php4 module has already been loaded, delete the AddModule line from httpd.conf

PHP4 Installation Under Linux

PHP is included with all current Linux distributions and is generally included in an Apache installation. You can check this with the command rpm -qa | grep -i php. Red Hat: If PHP is not yet installed, start the program redhat-config-packages. The packages are to be found in the group Webserver.

By default, the package php-mysql, which extends PHP with functions for communicating with MySQL, is not installed. Click on the button DETAILS in the group Webserver and select the package for installation.

Then Apache must be restarted.

root# /etc/init.d/httpd restart

SuSE: The two PHP packages mod_php4-core and apache2-mod_php4 are installed together with Apache 2. For PHP to function, the character string php4 must be inserted into the declaration APACHE_MODULES in the file /etc/sysconfig/apache2. (Possibly, this change will not be required in the final version of SuSE 9 and is a problem only with the beta version.)

#in the file /etc/sysconfig/apache2
...
APACHE_MODULES="access actions alias auth auth_dbm autoindex cgi
  dir env expires include log_config mime negotiation php4
  setenvif status suexec userdir"
...

Then relaunch Apache:

root# /etc/init.d/apache2 restart

Testing the PHP Installation

Create a test file phptest.php according to the following template, and save the file in a location where Apache can access it (that is, in the htdocs directory or in an alias directory). The name of the file should not be test.php, since Apache is frequently configured, for reasons of security, in such a way that this file is not evaluated.

<?php phpinfo(); ?>

Now load this page with a web browser. Be sure that the page is not read directly from the file system but via Apache. (For example, you must use http://localhost/phptest.php as the address.) The result should look like Figure 2-7.

Click To expand
Figure 2-7: PHP 4 runs!
Tip

If the test is unsuccessful, and instead of the table shown in Figure 2-7 you get only an empty page or a couple of lines of weird PHP code, the most probable cause of the trouble is that you are looking at the page directly and not via Apache. Please be sure that the path in the web browser contains http://localhost and not the direct file name, such as C:\path\phptest.php. Other possible causes of the error are Apache configuration problems. Have you restarted Apache after the change in the configuration file?

If you see nothing at all, and instead, a download dialog appears, then the Apache configuration has failed. Apache does not recognize the file type *.php, and therefore does not know that it is supposed to execute the PHP file.

If a download dialog continues to appear after a change in the configuration, then the web browser's cache might be at fault. In this regard, Mozilla seems especially guilty. Delete the cache to ensure that the problem is not caused by the web browser. (In Mozilla, execute EDIT |PREFERENCES |ADVANCED |CACHE |CLEAR XXX CACHE.)

Testing PHP in Combination with MySQL

The following program (see Figure 2-8) provides a list of all MySQL databases. (Immediately after installation there are only two databases: test is empty, while mysql is responsible for the MySQL internal management of access privileges.) The variable mysqlpasswd can remain empty as long as MySQL access is not secured by a password.

Click To expand
Figure 2-8: PHP can access MySQL.
<?php // mysql-test.php ?>
<html><head>
<title>Test PHP and MySQL</title>
</head><body>
<?php
  $mysqluser="root";           // user name
  $mysqlpasswd="xxx";         // password
  $mysqlhost="localhost";    // computer name
  $connID = mysql_connect($mysqlhost, $mysqluser, $mysqlpasswd);
  $result=mysql_list_dbs();
  echo "<p>Databases at the local MySQL server<p>\n";
  echo "<p>";
  while($row = mysql_fetch_row($result)) {
    echo "<br><i>$row[0]</i>\n";}
  echo "<p>";
?> </body></html>
Tip

If access to a database fails, either because the MySQL server is not running or because the user name and password are invalid, a series of error messages should appear (e.g., Warning: mysql_connect( ): Access denied for user: 'root@127.0.0.1' (Using password: NO).Inthiscase, set mysqluser and mysqlpasswd correctly.

If nothing but an empty page appears, then an error has probably occurred, but it is not displayed. Alter php.ini so that display_errors = On is valid (details in the next section).

PHP Configuration

Various configuration details of the PHP interpreter are controlled by the file php.ini. For this file to be recognized, under Unix/Linux it must be located in the directory /etc, while under Windows, in the PHP installation directory (e.g., c:\winnt\).

Under Linux, /etc/php.ini is usually preconfigured. In the Windows installation of PHP, on the other hand, you will find in the PHP installation directory several options for php.ini:

  • p.ini-recommended contains the settings recommended by the PHP developers, which are optimized for maximal security and speed.

  • p.ini-dist contains settings that are compatible, as far as possible, with earlier versions of PHP; in many PHP installations these settings are used by default.

You should copy one of these options into the Windows directory and change its name to php.ini. (PHP will work entirely without php.ini: Then the PHP default settings are valid.)

Warning

If PHP fails to function as it ought, if it presents indecipherable error messages (or none at all), then frequently, the trouble is with php.ini (or possibly also with an old version of php.ini from an earlier version of PHP!). In such cases, you should first create a backup copy of the existing php.ini file and then install php.ini-recommended into the etc directory or in the Windows directory.

Changes in php.ini become effective only after Apache has been restarted (if PHP is executed as an Apache module).

PHP Extensions Under Windows (xtensions)

PHP extensions make possible the use of additional functions that are not directly integrated into PHP. Which functions are directly integrated into PHP and which are realized as extensions depends on how PHP is compiled. The MySQL functions are generally directly integrated into PHP; that is, they do not have to be loaded as extensions. With PHP 4.3, a glance into the directory php-install-dir\extensions provides a good overview of the selection of available extensions.

If you are using PHP under Windows and wish to use PHP extensions, then the variable extension_dir must be set so that it specifies the directory of the PHP installation:

extension_dir = "c:\php-430\extensions\"

Furthermore, you must specify in php.ini the desired extensions via extension settings. The line for the OpenSSL module, for example, looks like this:

extension=php_openssl.dll

php.ini Settings for This Book

For my tests under Linux, I have not changed the file /etc/php.ini. For tests under Windows I have used php.ini-recommended as a basis for php.ini, and made only the following changes:

  • display_errors = On: this setting has the effect that error messages are displayed in the case of errors in PHP scripts. This is, of course, practical during program development. With servers deployed in the Internet, this setting should be Off, since error messages often give hints as to what is contained in the code and thus present a security problem. (If in developing a PHP page, instead of a result, only an empty page appears in the web browser, it is almost always the display_errors setting that is to blame.)

  • magic_quotes_gpc = On: This setting has the effect that in transferring data between web sites (Get/Post/Cookie), special characters, such as ', ", \, and a zero byte, are replaced by \', \", \\, and \0. Details on magic quotes can be found in Chapter 11. The setting has been chosen to be On, because that is the case with almost all Internet service providers.

  • default_charset: This option should either not be set at all (which is the case in the default setting), or set as "iso-8859-1". The setting default_charset controls whether in the transfer of HTML documents, the PHP interpreter should specify that the pages should be coded according to a particular character set. If default_charset is not set, the the coding of HTML documents can be set by a <meta> tag between <head> and </head>:

    <meta http-equiv="Content-Type"
      content="text/html; charset=iso-8859-1" />

The examples in this book generally use the ISO-8859-1 character set. This character set is also known as latin1, and it contains most of the characters used in Western European languages.

Tip

If you change php.ini, then take care that you do not change merely the comments, but the actual option settings. Comments in php.ini are indicated by a semicolon. Everything after a semicolon in a line is ignored!


phpMyAdmin

The best current tool for convenient administration of MySQL over an Internet connection is probably phpMyAdmin (Figure 2-9). The only condition for the use of phpMyAdmin is the installation of PHP on the computer on which the MySQL server is running. (Note that phpMyAdmin consists of numerous *.php files. Thus phpMyAdmin is not only a valuable tool, but at the same time an excellent example of MySQL programming with PHP.)

Click To expand
Figure 2-9: phpMyAdmin
Pointer

This section describes only the installation of phpMyAdmin on a local test system. Further information on other installation variants (such as for administration of MySQL by an Internet service provider) as well as an extensive description of the application possibilities of phpMyAdmin can be found in

Installation

You can find phpMyAdmin at the web site http://www.phpmyadmin.net/ as a *tar.gz or *.zip archive for downloading. All the files in this archive must be copied into a subdirectory of the DocumentRoot directory of Apache. (Depending on the operating system, the directory is called Programs\Apache Group\Apache[2]\htdocs, /var/www/html,or /srv/www/htdocs.

Under Windows, you can use WinZip, for example, for expanding the archive. Under Unix/Linux, on the other hand, you would use commands according to the pattern below (adapt the archive name and target directory as required). This book assumes that phpMyAdmin is installed in the directory phpmyadmin, but you can name the directory as you like:

root# tar xzf phpMyAdmin-n-php.tar.gz -C /srv/www/htdocs/
root# mv /srv/www/htdocs/phpMyAdmin-n /srv/www/htdocs/phpmyadmin

The hard disk space needed for phpMyAdmin 2.4 is about 5.5 megabytes.

Tip

With SuSE, phpMyAdmin is already included, and it can be installed with YaST. Asinstallation location the directory /srv/www/htdocs/phpMyAdmin is used.

Minimal Configuration for phpMyAdmin

Before you can use phpMyAdmin, you must first change some settings in the file config.inc.php in the phpMyAdmin installation directory. The listing presented below contains only the relevant lines for a minimal configuration. Normally, only the lines in boldface need to be changed.

The address must be contained in $cfg['PmaAbsoluteUri'], by which the phpMyAdmin files are reachable in the network (in my local testnetwork it is uranus.sol). Here localhost is not allowable! Instead of phpmyadmin, the directory under which phpMyAdmin is actually reachable in your installation must be given (in correct upper- and lowercase letters, and terminated with a forward slash).

You could attempt simply to leave $cfg['PmaAbsoluteUri'] empty; in many cases, phpMyAdmin succeeds in determining the correct setting itself. However, if that does not work, then in the use of phpMyAdmin, errors occur whose cause is difficult to trace:

# changes in config.inc.php
...
# must contain the computer name and the phpMyAdmin installation directory:
$cfg['PmaAbsoluteUri'] = 'http://uranus.sol/phpmyadmin/';
...

The following parameters specify how phpMyAdmin creates the connection to the MySQL server. If MySQL has not yet been secured with a password, then of course, the password string should remain empty:

$cfg['Servers'][$i]['host']       = 'localhost'; // MySQL hostname
. . .
$cfg['Servers'][$i]['auth_type']  = 'config';    // must be 'config'
$cfg['Servers'][$i]['user']       = 'root';      // MySQL user
$cfg['Servers'][$i]['password']   = 'xxx';       // MySQL password

Then you can test phpMyAdmin by placing in your web browser the address http://localhost/phpmyadmin/index.php or http://computername/phpmyadmin/index.php. The start page should look somewhat like that depicted in Figure 2-9 (depending on which version of phpMyAdmin you have installed and what access rights phpMyAdmin has on the MySQL server).

Controlling Access to the phpMyAdmin Directory with a Password

With phpMyAdmin you are faced with a large security problem: Not only you, but every web surfer in the known universe can manipulate your databases with phpMyAdmin, or even delete them (if the surfer discovers the directory in which you have installed phpMyAdmin). You can prevent this by securing access to the phpMyAdmin directory with a password via an .htaccess file. (This is not a MySQL password, but one for Apache! The use of .htaccess has been described already in the section in this chapter on Apache.)

Caution

A second security problem is presented by the file config.inc.php, which contains your MySQL password in plain text. Sometimes, the computer of the ISP is installed in such a way that web files can be accessed via FTP. then the protection offered by .htaccess is insufficient. Make sure that it is impossible to read config.inc.php by anonymous FTP.


Perl

Perl is perhaps the best-loved scripting language running under Unix. It is used by system administrators and web programmers (for CGI scripts), among others. However, Perl is of interest to you even if you have no intention of programming in Perl. Namely, there is a host of scripts for various MySQL administrative tasks. (Some are provided with MySQL, while many others can be found on the Internet.) In order for these scripts to be usable, Perl must be installed on your computer.

Remark

The examples for this book were tested with Perl 5.6 and 5.8. However, they should work without problem with earlier versions of Perl.

Installation

Perl Installation Under Windows

With its package ActivePerl, the company ActiveState offers the only current version of Perl for Windows. ActivePerl can be downloaded freely as an MSI file from www.activestate.com. (You will also find at that site a host of commercial Perl-related products.) With Windows ME or Windows 2000 you begin installation simply by double clicking on the MSI file. With older versions of Windows, you must first install the Windows installer (a download link can be found at www.activestate.com).

Perl Installation Under Linux

Perl is automatically installed with almost all Linux distributions, since countless administration tools require Perl. You can determine whether Perl is installed and, if so, what version, with the following command:

root# perl --version
This is perl, v5.8.0 built for i586-linux-thread-multi

If Perl happens not to have been installed, then you can install the Perl package with rpm or redhat-config-packages (Red Hat) or with YaST (SuSE).

Testing the Perl Installation

Create a test file with the name test.pl and the following contents:

#!/usr/bin/perl -w
print "Hello world!\n";
print "Please hit Return to end the program!";
$wait_for_return = <STDIN>;

Windows: Under Windows it should be possible to launch the file at once with a double click. The program prints "Hello world!" in a command window, waits for Return, and then terminates.

If a double click fails to bring about program execution, then open a command window and try the following command:

Q:\> perl Q:\directory_with_my_perl_files\test.pl

Linux: Under Unix/Linux you must first label test.pl as an executable file:

user$ chmod u+x test.pl

Now the program can be executed:

user$ ./test.pl
Hello World!
Please hit Return to end the program!<Return>

If this doesn't work, then perhaps the Perl interpreter has not been installed in the directory /usr/bin but in some other directory. Determine what directory this is, and make the necessary change in the file test.pl.

Remark

Under Unix/Linux it is not usual practice to give Perl files names of the form *.pl (even though it does no harm). Nonetheless, in this book we shall use this identifier to make it easier to exchange files between Windows and Unix/Linux.

MySQL Support for Perl

The installation of Perl is not the end of the matter. Perl is not capable of communicating with MySQL all by itself. Perl must be extended with two auxiliary modules:

  • DBI: DBI stands for database interface.With DBI modules we are dealing with a general interface for database programming.

  • DBD-Mysql: DBD stands for database driver. This module contains the MySQL-specific extension to DBI.

    Warning

    DBD-Mysql assumes that the MySQL client libraries are installed on the computer (files libmysqlclient and libmysql.dll).

    For Windows the client library is not available separately. Therefore, it is simplest to install the complete package. It is important that the PATH variable contain the directory with the file libmysql.dll (usually, C:\mysql\bin).

Installation of DBI and DBD-Mysql Under Windows

Extraordinarily frustrating was the attempt to install both packages under Windows (January 2003, ActivePerl 5.6.1 Build 633 and ActivePerl 5.8 Build 804). The installation takes place basically with the supplementary program ppm (Perl package manager), which is included with ActivePerl. The program can be started in a command window. It assumes a running Internet connection. The use of ppm follows from the following code lines (for ActivePerl 5.6.1):

C:> ppm
PPM interactive shell (2.1.6) - type \sqhelp\sq for available commands.
PPM> install dbi
Install package \sqdbi?\sq (y/N): y
Installing package \sqdbi\sq . . .
Bytes transferred: 401795
Installing C:\Programs\perl\site\lib\auto\DBI\dbd_xsh.h
Installing C:\Programs\perl\site\lib\auto\DBI\DBI.bs
. . .
Writing C:\Programs\perl\site\lib\auto\DBI\.packlist
PPM> install DBD-Mysql
Install package \sqDBD-Mysql?\sq (y/N): y
Installing package \sqDBD-Mysql\sq . . .
Error installing package \sqDBD-Mysql\sq: Could not locate a PPD file
for package DBD-Mysql

ActivePerl 5.8 contains a newer version of ppm. The command input is the same as with the previous version, though the displayed information looks a bit different.

Tip

If your Internet connection is via an HTTP proxy, then before the install command, the following command must be executed, where myproxy.com:8080 should be replaced by the actual proxy address:

set http_proxy=http://myproxy.com:8080/
Tip

You can obtain the available packages with search <pattern>. For example, search mysql finds all supplementary Perl packages whose name contains mysql (regardless of where in the name the string is located).

The problem here is that ppm does not find the package dbd-mysql either with ActivePerl 5.6.1 or ActivePerl 5.8 (see the error message at the end of the command install dbd-mysql). Here is a solution:

  • If you are lucky, this problem will have been solved by the time this book has appeared in your hands. Then dbd-mysql should function as it did previously.

  • If youare working with ActivePerl 5.6.1, you can use the following command for installation. Note the upper- and lowercase usage!

    PPM> install http://www.activestate.com/PPMPackages/5.6/DBD-Mysql.ppd
Install package \sqhttp://www.activestate.com/PPMPackages/5.6/
  dbd-mysql.ppd?\sq (y/N): y
Installing package \sqdbd-mysql\sq ...
Bytes transferred: 104405
Installing C:\Programs\perl\site\lib\auto\DBD\mysql\mysql.bs
Installing C:\Programs\perl\site\lib\auto\DBD\mysql\mysql.dll
...
Writing C:\Programs\perl\site\lib\auto\DBD\Mysql\.packlist

    Basically, the installation works. The problem is that in this manner you end up installing an ancient version of dbd-mysql from June 2000.

  • If you are working with ActivePerl 5.8, then installation works with the following command (according to the discussion thread in the mailing list mailing.database.msql-mysql-modules; see http://www.mysqlforums.com/):

    PPM> install http://theoryx5.uwinnipeg.ca/ppms/DBD-mysql.ppd

    The problem this time is one of stability. Perl programs with MySQL access are executed, but at the program's end the Perl interpreter crashes or remains in an endless loop. (Hit Control+C!) It is not clear whether these problems are related to the particular version (4.0.n)of the MySQL server.

  • With both ActivePerl versions, you can install the package DBD-mysqlPP instead of DBD-Mysql. This is a relatively new implementation of DBD-Mysql, which does not depend on the MySQL client library. Therefore, the package can be used even if the MySQL client library is not installed or (in the case of some operating systems) not even available.

    Unfortunately, DBD-mysqlPP is not fully mature and also not completely compatible with DBD-Mysql. Thus there are necessary code changes in Perl scripts that were originally planned for DBD-Mysql.Some DBD-Mysql functions are not implemented at all. Finally, DBD-mysqlPP is said to be slower than DBD-Mysql (which I have not checked).

    In short, DBD-mysqlPP is currently an imperfect substitute for DBD-Mysql.

In the end, I made my experiments under Windows with ActivePerl 5.6.1 and the old version of DBD-mysql for Perl 5.6.

Installation of DBI and DBD-Mysql Under Linux

Red Hat: With Red Hat 9.0, the packages perl-DBI and perl-DBD-MySQL are located on CD 2. The installation is accomplished most easily with rpm -i package file name. SuSE: With SuSE you install the packages perl-DBI and perl-Msql-MySQL-modules with YaST2.

With both distributions you can check whether the packages are already installed with the command rpm-qa | grep -i perl.

Testing DBI and DBD-Mysql

The following Perl script shows all databases available to the MySQL server. Immediately after installation these are only mysql and test. The example will work only if the given user name and password permit access to this table. Change this information as need be. (The words in question are in slant script. If MySQL is still not password protected, then both character strings can be left empty. Information for setting MySQL access privileges can be found in Chapter 9.)

#!/usr/bin/perl -w
# Test file mysql-test.pl
use DBI();
# establish connection to database
$dbh = DBI->connect(
  "DBI:mysql:database=mysql;host=localhost",
  "user", "password", {'RaiseError' => 1});
# execute query
$sth = $dbh->prepare("SHOW DATABASES");
$sth->execute();
# show results
while(@ary = $sth->fetchrow_array()) {
  print join("\t", @ary), "\n";
}
$sth->finish();

The result of the script will look something like this (depending on how many databases have already been installed):

mysql
test

Apache Configuration for Perl and CGI

If you wish to use Perl via CGI for programming dynamic HTML pages, then you must store the Perl files in the directory cgi-bin of the Apache web server. The default location of this directory is system-dependent:

Windows | Program Files/Apache Group/Apache/cgi-bin
Red Hat Linux | /var/www/cgi-bin
SuSE Linux | /srv/www/cgi-bin

Furthermore, Apache must be so configured that Perl scripts are executed by the Perl interpreter. For this there are two possibilities: Either the Perl interpreter is restarted for every script, or the interpreter remains in memory. The second variant is much more efficient, but it assumes the Apache module mod-perl.

CGI on Windows Computers

For Apache actually to execute the page http://www.meine-firma.com/cgi-bin/hello-cgi.pl, the following setting within httpd.conf is necessary:

# in httpd.conf
ScriptAlias /cgi-bin/ "C:/Program Files/Apache Group/Apache/cgi-bin/"
<Directory "C:/Program Files/Apache Group/Apache/cgi-bin">
   AllowOverride None
   Options None
   ScriptInterpreterSource registry
</Directory>

The parts in slant font must, of course, be replaced by the directories that will actually be used on your computer.

Remark

Pay particular attention to the ScriptInterpreterSource line, which is not always present by default. This line is necessary, because Apache normally evaluates the first line of a Perl script in order to search for the Perl interpreter. This line contains, as a rule, the text #!/usr/bin/perl. Under Unix/Linux the interpreter is located there, but not under Windows.

With the ScriptInterpreterSource line you obtain the result that Apache determines the location of the Perl interpreter, based on the file identifier of the script (which must now be *.pl) and Windows registry. If you forget ScriptInterpreterSource, then you will generally get the error message Internal Server Error.

Note also that apache must be restarted after a change in httpd.conf.

CGI on Unix/Linux Computers

With Red Hat and SuSE, httpd.conf is already correctly configured and does not need to be altered.

  • Special for Red Hat: By default, all necessary modules are already installed.

  • SuSE: To execute the example programs for this book you need the package perl-HTML-Parser, which in general you will have to install. All other Perl packages should be available by default.

  • Access privileges: Note that the files in the cgi-bin directory are executable and that Apache (which usually is executed under a special account such as apache or wwwrun) is also allowed to read these files. Use chmod to set the appropriate access privileges.

    Tip

    Even if you move the CGI files via FTP into the cgi-bin directory of your ISP, you must set the access privileges. The FTP protocol provides for this possibility, and most FTP programs are capable of doing so. (With the program WS_FTP, which runs under Windows, you can click on the files with the right mouse button and execute the command chmod.)

Testing CGI Scripts

To test whether CGI files are executed, create the file cgi-test.pl in the cgi-bin directory:

#!/usr/bin/perl -w
# Test file cgi-test.pl
use CGI qw(:standard);
use CGI::Carp qw(fatalsToBrowser);
print header(),
  start_html("Hello CGI"),
  p("Hello CGI"),
  end_html();

If you are working under Unix/Linux, then you must set the access privileges with chmod g+rx or chmod a+rx in such a way that Apache is able to access the files. Then open the following page with a web browser (see Figure 2-10). Instead of www.myserver.com, you must specify the network address of your computer or localhost:

http://www.myserver.com/cgi-bin/cgi-test.pl
Click To expand
Figure 2-10: CGI test page


Connector/ODBC

ODBC (Open Database Connectivity) is a popular mechanism, particularly under Windows, for uniform access to a wide variety of database systems. The only requirement is an ODBC driver for the database system, which creates the interface between the ODBC system and the database.

Connector/ODBC 3.51 (formerly MyODBC) is the ODBC driver for MySQL. The driver supports the functions defined in ODBC version 3.5.1 (according to the on-line handbook, Complete Core API and Level 2 Features). For installation you should obtain Connector/ODBC from the MySQL web server. Most recently, Connector/ODBC was to be found at the following address:

http://www.mysql.com/downloads/api-myodbc-3.51.html

There are two different download variants:

  • Connector/ODBC with setup program (driver installer)for initial installation;

  • Only the driver DLLs (DLLs only) for an update.

For initial installation use the setup program. If later you wish to update the installation, it suffices to replace the file myodbc3.dll in the Windows system directory (e.g., C:\Windows\System32). There is no provision for a deinstallation of MyODBC.

Remark

ODBC is now also used under Unix. Thus there exists Connector/ODBC for Unix. However, in this book we treat only the Windows version of Connector/ODBC.

Setup Program

Remark

The MySQL documentation announces that the Connector/ODBC installation under Windows sometimes breaks down to the accompaniment of mysterious error messages (for example, that the file Windows\System\Mfc30.dll cannot be copied). It is recommended as a solution to this problem to start up Windows in safe mode. To activate this mode it is necessary at Windows startup to hold down the F8 key. This should lead to a successful installation. I must say that I have installed MyODBC successfully on a variety of computers without having to take this measure.

Connector/ODBC Character Set

Connector/ODBC does not assume a MySQL client installation on the computer. If you use ODBC on a computer exclusively for access to a MySQL server running somewhere else, then the installation of Connector/ODBC alone suffices.

In Connector/ODBC, only the default character set of the client tools is included. (This includes latin1, big5, euc_kr, gb2312, gbk, sjis, tis620, and ujis. If the server uses a different character set (option default-character-set), then one receives the error message Can't initialize character set n.

The solution to this problem is simple: The directory c:\mysql\share\charsets must be created on the the ODBC computer. There, all MySQL character set files must be copied. (These files reside on the computer on which the MySQL server is running. If it is a Windows machine, the path is usually Program Files\mysql\share\charsets, while under Unix/Linux it might be /usr/local/mysql/share/mysql.)


Editors

To complete the testing environment you will need an editor for developing your MySQL programs. If you are developing web applications and are using, say, PHP or Perl, in principle you can use any editor you like, under Windows even notepad.exe. It is more convenient to work with an editor that understands HTML/PHP/Perl, etc., that displays your code in different colors, assists with the insertion of frequently used code blocks, and so on. Under Linux, the KDE standard editor kate is a good choice.

Tip

The choice of editor is, of course, primarily a matter of taste. But if you are looking for a good editor, the links at my web site, www.kofler.cc/mysql, might be of use. Popular editors include Quanta and Bluefish (Linux) and HTML-Kit (Windows).

Emacs/XEmacs

The editors Emacs and XEmacs are considered by many Unix/Linux fans to be the measure of all things. Their use takes a bit of getting used to, but in exchange these editors are probably unsurpassed for their adaptability to your individual requirements, and they are more stable than most other editors. (X)Emacs is obtainable without charge for Unix/Linux and is also obtainable for Windows. (As the names suggest, these two editors have a common ancestor. They are relatively compatible one with the other and differ only a little in their use.)

http://www.gnu.org/software/emacs/emacs.html
http://www.xemacs.org/

Both Emacs and XEmacs work without problems with HTML files as well as with files of most programming languages (Perl, Java, C, etc.). They can highlight different code elements in color, help with compiling, etc. However, the situation with PHP support is less good. There is no PHP mode included with the current versions of Emacs and XEmacs. The simplest solution is to select C++ mode. (This, however, does not work properly with HTML code, as one would expect.) To make Emacs/XEmacs view a PHP file as a C++ file, add the following text to the first line:

<?php // -*- C++ -*- --

On the Internet one can find certain PHP extensions for Emacs. However, their installation requires a fundamental knowledge of the internal workings of Emacs. Links can be found at my web site.

vi/vim

Unix fans who dislike the editor (X)Emacs usually use vi or vim. This editor has similar functions and modes to those of (X)Emacs, but takes even more getting used to.


Using MySQL with an ISP

This book generally assumes that you have installed MySQL on your own computer. It is only in this way that you have the possibility of trying out MySQL in all of its many facets. Furthermore, it is helpful to develop MySQL solutions first on a test system and to port them to a web site only after sufficient testing.

However, in principle, it is also possible to do without one's own MySQL installation entirely. In this case you develop a dynamic web site based exclusively on the MySQL access that your Internet service provider makes available. Here we shall briefly describe the necessary course of action.

MySQL Administration at an ISP with phpMyAdmin

As a rule, ISPs permit only local MySQL access for reasons of security. You can thus register with MySQL only if you are working locally on a computer of the ISP. But generally, that is precisely what you are not allowed to do (it could also be that you have telnet/ssh access, but that is also seldom allowed due to security considerations).

To the extent that your ISP supports PHP, MySQL administration is still possible. The favored program for this purpose is the PHP script collection phpMyAdmin that we have already mentioned. These scripts fulfill the condition that they can be executed on the ISP computer, while on the other hand, phpMyAdmin can be conveniently used over the Internet (that is, for administration you simply use a browser that runs locally on your computer).

With a little luck, your ISP offers phpMyAdmin. If that is not the case, then you must install phpMyAdmin yourself. If you do not have telnet/ssh access, first unpack the phpMyAdmin archive on your local computer. Then set the access information in config.inc.php (ISP host name, user name, password). You should have obtained this information from your ISP.

Finally, use an FTP client (under Windows, for example, WS_FTP) to move all the phpMyAdmin files from your computer into a WWW directory (such as phpmyadmin) at your ISP.

If all has gone well, then you can now launch phpMyAdmin with a browser with the address www.mysite.com/phpmyadmin/index.php or -/index.php3. With phpMyAdmin you can now create and delete your own database tables, add data, etc. You may also in some circumstances be able to use phpMyAdmin for the purpose of creating new databases. That depends on the MySQL administration privileges granted you by your ISP. (Details on config.inc.php as well as further information on using phpMyAdmin can be found in Chapter 4.)

Caution

Everyone who guesses the address of phpmyadmin/index.php can administer your database just as you can. It is thus absolutely essential that access to the directory phpmyadmin be secured via a .htaccess file. The method for doing this is described in the section on Apache in this chapter.

Database Upload

Pointer

If you develop MySQL solutions first locally and then port them to the computer of your ISP, it is necessary to create there a copy of your test database. Thus you must copy the locally created database (perhaps together with all its data) to the MySQL server of the ISP. This modus operandi is often called "database upload." Hints on how to carry out this upload can be found in the last section of Chapter 10.


Chapter 3: Introductory Example (An Opinion Poll)

Overview

The best way of becoming familiar with a new database or development system is to work through a full-fledged example. Thus in this chapter our goal is to create a web site for the purpose of conducting an opinion poll.

To a certain extent this is a trivial example, and its results certainly could be accomplished without the use of MySQL. However, it brings into focus the interplay between MySQL and a script programming language (for this example we have used PHP). Moreover, our example casts light on the entire process of database design from first beginnings right up to the completed application.

Our opinion poll consists of two pages. The file vote.html contains a simple questionnaire with a single question: What is your favorite programming language for developing MySQL applications? The question is followed by a selection of choices for a response (see Figure 3-1). After one of the options is selected and OK is clicked, the result page results.php is displayed (see Figure 3-2).

Click To expand
Figure 3-1: The questionnaire
Click To expand
Figure 3-2: Results of the survey

Assumptions

You can try out this example on my web site (www.kofler.cc). However, from a pedagogical point of view it would be better for you to attempt to recreate this example for yourself. For this you will need a test environment consisting of Apache/MySQL/PHP that permits you the following:

  • creation of a new MySQL database (that is, you need sufficient privileges to be able to execute CREATE DATABASE)

  • moving files into a directory on the web server

  • executing PHP script files

Information on setting up such a test environment on a local computer can be found in the previous chapter.

A complete understanding of our example requires some basic knowledge of databases. If you have never had much, if anything, to do with databases and as a consequence run into difficulties in understanding what is going on, please do not despair. Chapter 5 provides an introduction to relational database systems, while Chapter 6 explains how to use the database query language SQL.

In this example, the programming language PHP will be used. The code is rather straightforward; that is, you should have no difficulty in understanding it even you don't know a word of PHP. However, you should know in general how embedded script languages function in HTML. (The Active Server pages developed by Microsoft are based on the same idea.)


Database Development

To save the results of our questionnaire with MySQL, you must first set up a database and then place a table in that database. (Every MySQL database consists of tables. A particular feature of this example is that only a single table is required. As a rule, that is, when the requirements of the project are more complex, several linked tables will be used.)

Executing mysql

Both operations—generating a database and creating a new table—require that you communicate with MySQL. Under Unix/Linux you execute the command mysql. Under Windows you search in Explorer for the program mysql.exe and launch it with a double click. (The program should reside in the bin directory of the MySQL installation directory, that is, in C:\Programs\MySQL\bin.)

Remark

If mysql immediately terminates, possibly with an error message such as Access denied for user xy, then either access to MySQL is completely denied to user xy, or it is protected by a password. In either case you must invoke mysql with the options -u name and -p:

> mysql -u username -p
Enter password: xxx

Furthermore, mysql has dozens of other options. The most important of these are described in Chapter 4, while all of them are collected in Chapter 19. Background information on the MySQL security system (access protection, passwords, privileges) can be found in Chapter 9.
Remark

The distinction between MySQL and mysql is somewhat confusing. MySQL denotes the database server, which normally is launched automatically at system startup. The server runs continuously in the background. (This chapter assumes that the server is up and running.) The program name of the server is, depending on the operating system, mysqld (Unix/Linux) or mysqld.exe, mysqld-nt.exe, etc. (Windows).

In contrast to these, we have the auxiliary programs mysql and mysql.exe.These programs come into play when administration or maintenance are to be carried out interactively. The program mysql has the task of transmitting interactive commands to the server and displaying the results of these commands on the monitor. The official name of mysql is MySQL Monitor, but the functionality and mode of action are more reminiscent of a command-line interpreter.

Alternatives to mysql, such as the more convenient HTML interface phpMyAdmin, will be introduced in Chapter 4.

In mysql you can now input commands that will be transmitted to the database server (see Figure 3-3). To test whether a connection can even be made, execute the command STATUS. The result should be the display of various pieces of status information about the database (such as the version number):

mysql>  STATUS;
mysql Ver 12.18 Distrib 4.0.10-gamma, for Win95/Win98 (i32)
Current user:           ODBC@localhost
SSL:                    Not in use
Server version:         4.0.10-gamma-nt
Protocol version:       10
Connection:             localhost via TCP/IP
Client characterset:    latin1
Server characterset:    latin1
TCP port:               3306
Uptime:                 12 days 1 hour 26 min 20 sec1
Threads: 2   Questions: 3099   Slow queries: 0   Opens: 6
Flush tables: 1 Open tables: 0   Queries per second avg: 0.003
Click To expand
Figure 3-3: The MySQL monitor
Pointer

If problems arise in starting up mysql or in executing status, the most probable cause is that the database server hasn't even been started or that access has been denied to you. More information on installation can be found in Chapter 2, while information on access and security can be found in Chapter 9.

Setting Up the Database

To set up the new database test_vote, in mysql execute the command CREATE DATABASE. Note that you must end the command with a semicolon. In the following two lines your input appears in boldface:

mysql> CREATE DATABASE test_vote;
Query OK, 1 row affected (0.01 sec)

The reply delivered by mysql may look a bit weird. The output 1 row affected indicates that in the list of all databases, which internally, of course, is in the form of a MySQL table, one row was changed. What is important here is only that the CREATE DATABASE command was executed correctly.

Remark

The database name test_vote was not chosen quite arbitrarily. In the default setting of MySQL access privileges, every user is permitted to create databases on a local computer that begin with the word "test." In particular, when you yourself are not the MySQL administrator (but rely on the help of a system administrator), a name of the form test_xy can save any number of e-mails or telephone calls.

The drawback of the name test_xy is that every user of the local computer can edit, or even delete, the database. This is no problem for this introductory example, but in the case of a real-life application you probably will want to give your database abitmore security. Necessary information on this can be found in Chapter 9.

Creating Tables

The database test_vote has been created, but it is not yet possible to store any information. For this you need tables. To create a new table within the database test_vote, use the command CREATE TABLE.

Before you execute this command, however, you must specify the database into which the table is to be placed. The requisite command for this is USE. It determines the default database to which further commands are to be applied. (MySQL is managing other databases besides the newly created database test_vote.)

mysql> USE test_vote;
Database changed
mysql> CREATE TABLE votelanguage (
    -> id INT NOT NULL AUTO_INCREMENT,
    -> choice TINYINT NOT NULL,
    -> ts TIMESTAMP,
    -> PRIMARY KEY (id));
Query OK, 0 rows affected (0.01 sec)

The CREATE TABLE command may seem a bit strange at first. But just go ahead and input the boldface commands listed above line for line. (You can terminate each command with Return. The semicolon indicates to mysql that the end of a command has been reached.)

Tip

If you should make a typing error during the input of a command, MySQL will usually inform you of this fact with an error message. You must now repeat the entire command. Using the cursor keys á and ¯ you can recall and correct previously input lines.

If MySQL has accepted CREATE TABLE in spite of a typographical error (because the command, though semantically not what you had in mind, is nonetheless syntactically correct), you can delete the incorrectly defined table with DROP TABLE votelanguage;. That accomplished, you can repeat the command CREATE TABLE.

Instead of creating the table votelanguage via a CREATE TABLE command, you could, of course, use an administration tool such as MySQL ControlCenter or phpMyAdmin. This assumes, of course, that such a program is installed and that you have learned how to use it. The program mysql is not known for its ease of use, but it has the advantage of offering the best way of documenting the steps to be executed.

Now let us explain what is actually going on. With CREATE TABLE you have brought into being a table with three columns, id, choice, and ts.Once the table is filled with data (namely, the results of the survey), the content of the table can be displayed something like this:

id     choice     ts
1      4          20030114154618
2      5          20030114154944
3      4          20030114154953
4      3          20030114154954
5      3          20030114154957
6      6          20030114155012
7      3          20030114155021
8      1          20030114155027
...

The interpretation of these data is that the first person to respond to the survey chose the programming language PHP, while the second chose VB, and the third selected PHP. The next respondents chose, in order, Perl, Other, Perl, and C. The column id thus contains a running identification number that identifies the lines (the data set). The column choice contains, in coded form, the selection made by the survey participant, where the numbers 1 through 6 correspond to the programming languages C, Java, Perl, PHP, VB, and Other. The column ts contains the time at which the query was executed. (For example, the first participant filled out vote.html on January 14, 2003, at 15:46:18.)

Tip

Like any other specialized subject, the database world has its own argot. Database vocabulary includes the term data record for each line of the table above. Instead of columns (here id and choice) oneoften speaks of fields.

In order to generate a table with the two columns id and choice, the following command would suffice:

CREATE TABLE votelanguage (id INT, choice TINYINT, ts TIMESTAMP);

The result is that the column id is declared with the data type INT, and choice is declared to be of data type TINYINT. This means that in theory, 231 individuals (2,147,483,648, that is) could participate in our survey before the range for id was exhausted. (If id were declared as type UNSIGNED INT, then the number of potential participants would be doubled.) In choice, on the other hand, there are 216 different values available. (The data types INT and TINYINT are discussed in Chapter 5 together with the other MySQL data types.)

By this point, you may be wondering why I have plagued you with such a complicated command as CREATE TABLE when there is a much easier way to achieve the same result. The difference between the complicated and simple variants is the difference between good and bad database creation. (And you wouldn't want me to be leading you astray on our very first example, would you?)

The attribute AUTO_INCREMENT for the column id has the effect that with each new record the appropriate value for id is automatically inserted. Thus when the results of the survey are saved, only choice has to be specified, since the database takes care of id on its own. This attribute ensures a consistent numbering of the data records, which is important for efficient management of the table.

The attribute NOT NULL ensures that actual values must be placed in both columns. It is not permitted to store the data record NULL or (in the case of choice) not to insert any value at all. Thus this attribute prevents invalid data records from being stored. (Go ahead and try to force MySQL to accept such a data record. It will refuse and present you with an error message.)

PRIMARY KEY (id) has the effect that the column id is used to identify the data records. That is the reason that the column was provided for in the first place, but MySQL is not so clever as to be able to read your mind, and it requires that it be informed precisely as to what your wishes are. (In this case, id will from now on be called a primary key.) The definition of a primary key has a decisive influence on the speed with which data records can be accessed. That holds especially for linked tables—but let us not get ahead of ourselves. We shall stick for the moment with simple tables, always define a single primary key (and do so, if possible, for an INT field with the attributes NOT NULL and AUTO_INCREMENT).

Why Make It Complicated, When It Could Be So Much Easier?

Perhaps it has occurred to you that we have presented a rather complex solution to an easy problem. It could done in a much simpler fashion. All we need to do is to control six counters, such as in a table of the following form:

id     counter
1      2
2      0
3      7
4      9
5      2
6      1

Such a display would mean that two participants expressed a preference for the C language, none for Java, seven for Perl, nine for PHP, and so on. Each time a preference is registered, the corresponding counter is incremented by 1. The database would consist altogether of six lines; one wouldn't have to worry about performance; memory requirements would be essentially zero—nothing but advantages! And all of this is aside from the fact that the six counters could be effortlessly stored in a small text file. In short, MySQL is totally unnecessary for this survey.

All right, then, you are correct! However, such an attitude is helpful only as long as we are dealing with a simple application. What happens when you would like to offer each participant the opportunity to make a comment? What if you allow the participants to fill out the questionnaire only after a login (to rule out the possibility of multiple voting)? What if you wish to record the time and IP address for each participant (again to secure against attempts at manipulating the survey)?

In all these cases, you would have to store the responses in a table like the one that we have presented. (The table would have merely to be enlarged by a column or two. The structure of the program can remain essentially the same.) So, if the conception of this example seems a bit overly complicated at first glance, the reason is that we are keeping open the possibility, even in this first example, of a later extension.


The Questionnaire

As we have mentioned already in the introduction, our entire project consists of two web pages. The first page (vote.html) contains the questionnaire. This is pure HTML code (without PHP). The second page (results.php) carries out two tasks: evaluating the questionnaire and displaying the results.

The HTML code of the questionnaire is given here, so that the code for evaluation in the next section will be understandable. Most important are the attributes name and value of the elements of the questionnaire. All of the radio buttons have the name vote, while value contains values between 1 and 6. The OK button has the name submitbutton, and as value the character string "OK" is used.

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN">
<!-- php/vote/vote.html  -->
<html><head>
    <meta http-equiv="Content-Type"
    content="text/html; charset=iso-8859-1" />
    <title>MySQL-poll</title>
</head>
<body>
<h2>MySQL- poll </h2>
<p><b> What is your favorite
programming language for developing MySQL applications?</b>
<form method="POST" action="results.php">
    <p>
    <input type="radio" name="vote" value="1">C/C++
    <br><input type="radio" name="vote" value="2">Java
    <br><input type="radio" name="vote" value="3">Perl
    <br><input type="radio" name="vote" value="4">PHP
    <br><input type="radio" name="vote" value="5"> ASP[.NET] / C# / VB[.NET] / VBA
    <br><input type="radio" name="vote" value="6"> Other
    </p>
    <p><input type="submit" name="submitbutton" value="OK"/></p>
</form>
<p>Go directly to the <a href="./results.php">results</a>.</p>
<p>Back to the
<a href="../mysqlbook.html">MySQL book page</a>.
</body>
</html>


Questionnaire Evaluation and Displaying Results

One can call results.php either directly (for example, via a link) or using the data in the questionnaire. In one case, what is shown is only the current state of the questionnaire. In the other case, the data are also evaluated and stored in the database.

Establishing a Connection to the Database

The opening lines of results.php consist of HTML code. Then comes the first task within the PHP code, which is to create a link to MySQL. For this purpose, the PHP function mysql_connect is used, to which three pieces of information are passed:

  • user name

  • password

  • computer name (host name)

If you have installed MySQL yourself on the local computer and have not yet secured it with a password, then you may simply pass empty character strings as parameters. The specification of the computer name is necessary only if the web server (that is, Apache) is running on a different computer from the one that is running MySQL.

The function mysql_connect returns an identification number that is stored in the variable link.In the present example, link is used only to identify possible problems in creating the link. In this case, an error message is displayed, and the PHP code is ended abruptly with exit. Otherwise, the active database is selected with mysql_select_db. (MySQL usually manages several databases, and the effect of mysql_select_db is that the following commands automatically refer to test_vote.)

Evaluating the Data and Storing Them in the Database

The next task in results.php consists in evaluating the data in the questionnaire and storing them in the database. The data are passed to the elements submitbutton and vote of the global field $_POST.(Compare the name and value attributes in vote.html.) The auxiliary function array_item is used to read out this field.

Warning

If you try out this example with PHP 4.0 or an earlier version of PHP, you must comment out or delete the lines $submitbutton = array_item( ... ) and $vote =array_item( ... ).

The field $_POST is not available in older versions of PHP. Instead, the variables $submitbutton and $vote are defined automatically. However, this behavior is relatively insecure and was therefore changed beginning with PHP 4.1.

If results.php was called via the questionnaire of vote.html, then the variable submitbutton contains the character string "OK".

For the sake of security, a validation test is also carried out for the variable vote. (For example, a participant may have forgotten to select one of the choices.) If vote contains a valid value, then this value is stored in the database. The SQL command to accomplish this will look something like the following:

INSERT INTO votelanguage (choice) VALUES (3)

What INSERT INTO accomplishes is to insert a new data record (a new row) into the table votelanguage. The expression (choice) specifies all of the fields of the data record for which values should be passed by means of the command VALUES ( ... ). Since MySQL takes responsibility on its own for the field id (attribute AUTO_INCREMENT; see above), in this example choice is the only field affected. Of course, instead of "3" the value of the variable vote will be placed, depending on which of the programming languages was selected in the questionnaire. The SQL command thus constructed is then passed along to MySQL with the PHP function mysql_query.

Pointer

If you have never had dealings with SQL and thus are unfamiliar with the syntax of SQL commands, fear not. In Chapter 6 you will find an extensive introduction to SQL. The two commands used in this chapter, namely, INSERT and SELECT,arepresumably more or less self-explanatory.

The columns id and ts of the votelanguage table are automatically filled in by MySQL: id with the unique running number, and ts with the current time.

Displaying the Survey Results

Regardless of whether the questionnaire has just been evaluated, the previous results of the survey must be able to be displayed. (If a new vote has been cast, it will be taken into account.)

First a check must be made as to whether the votelanguage table contains any data at all. (When the questionnaire is first placed on the Internet, no votes have yet been cast.) The required SQL query looks like this:

SELECT COUNT(choice) FROM votelanguage

The SQL command is again executed with mysql_query. What is new this time is that a link to the result of the query is stored in the variable result. The result of a SELECT command is, in general, a table. However, in the above example this table consists of merely a single row and a single column. (Furthermore, result contains only an ID number, which is used as a parameter in various other mysql_xxx functions. It is the task of PHP to take care of the actual management of the result.)

To evaluate the result, the function mysql_result($result, 0, 0) is used, by which the element in the table from the first row and first column is read. (With MySQL functions, counting begins with 0.)

Provided that the votelanguage table is not empty, a loop is executed to report the percentage of votes cast for each programming language. The requisite SQL queries look similar to the one above, only now the number of data records that contain a particular value of choice (for example, 3)iscounted:

SELECT COUNT(choice) FROM votelanguage WHERE choice = 3

For evaluation, mysql_result is used. Then a bit of calculation is necessary to round the percentages to two decimal places.

Program Code (results.php)

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN">
<!-- php/vote/results.php-->
<html><head>
<meta http-equiv="Content-Type"
                content="text/html; charset=iso-8859-1" />
<title>Survey Result</title>
</head><body>
<h2> Survey Result </h2>
<?php
    $mysqlhost="localhost";
    $mysqluser="root";
    $mysqlpasswd="";
    $mysqldbname="test_vote";
    Create Link to Database
    $link =
        @mysql_connect($mysqlhost, $mysqluser, $mysqlpasswd);
    if ($link == FALSE) {
        echo "<p><b>Unfortunately, no link to the database
                can be made. Therefore, the results
                cannot be displayed at present. Please try
                again later.</b></p>
                </body></html>\n";
        exit();
    }
    mysql_select_db($mysqldbname);
    // if questionnaire data are available:
    // evaluate + store
    function array_item($ar, $key) {
        if(array_key_exists($key, $ar)) return(\ar[\key]);
        return(\sq\sq);   }
    $submitbutton = array_item($_POST, \sqsubmitbutton\sq);
    $vote = array_item($_POST, \sqvote\sq);
    if($submitbutton=="OK") {
        if($vote>=1 && $vote<=6) {
                mysql_query(
                "INSERT INTO votelanguage (choice) VALUES ($vote)");
        }
        else {
    echo "<p>Not a valid selection. Please vote
              again. Back to
              <a href=\" vote.html\">questionnaire</a>.</p>
              </body></html>\n";
    exit();
        }
    }
    // display results
    echo "<P><B> What is your favorite programming language
                for developing MySQL applications?</b></p>\n";
    // Number of votes cast
    $result =
        mysql_query("SELECT COUNT(choice) FROM votelanguage");
    $choice_count = mysql_result($result, 0, 0);
    // Percentages for the individual voting categories
    if($choice_count == 0) {
        echo "<p>No one has voted yet.</p>\n";
    }
else {
        echo "<p>$choice_count individuals have thus far taken part
                in this survey:</p>\n";
        $choicetext = array("", "C/C++", "Java", "Perl", "PHP",
                "VB/VBA/VBScript", "Other");
        print("<p><table>\n");
        for($i=1; $i<=6; $i++) {
    $result = mysql_query(
                "SELECT COUNT(choice) FROM votelanguage " .
                "WHERE choice = $i");
    $choice[$i] = mysql_result($result, 0, 0);
    $percent = round($choice[$i]/$choice_count*10000)/100;
    print("<tr><td>$choicetext[$i]:</td>");
        }  print("<td>$percent
        print("</table></p>\n");
    }
?>
</body>
</html>

The Resulting HTML Code

If you are relatively inexperienced with PHP, you might find it helpful to have a look at the resulting HTML code (that is, what the user finally sees in the browser, as depicted in Figure 3-2) as an aid to understanding the program presented above.

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN">
<!-- results.php -->
<html><head>
<meta http-equiv="Content-Type"
    content="text/html; charset=iso-8859-1" />
<title>Survey Results</title>
</head><body>
<h2>Survey Results</h2>
<p><b>What is your favorite programming language
          for the development of MySQL applications?</b></p>
<p>17 people have thus far taken part
          in this survey:</p>
<p><table>
<tr><td>C/C++:</td><td>17.65 %</td></tr>
<tr><td>Java:</td><td>11.76 %</td></tr>
<tr><td>Perl:</td><td>17.65 %</td></tr>
<tr><td>PHP:</td><td>41.18 %</td></tr>
<tr><td>ASP[.NET] / C# / VB[.NET] / VBA:</td><td>11.76 %</td></tr>
<tr><td>Other:</td><td>0 %</td></tr>
</table></p>
</body></html>


Ideas for Improvements

Layout

It may have occurred to you that I have not paid much attention to the appearance of things in this example. Naturally, the results of the survey could be presented in a colorful bar graph. However, that has nothing to do with the subject of this book, namely, MySQL. Introductions to the attractive presentation of HTML documents, to the PHP programming of graphics libraries, and the like can be found in quantity in a number of books on HTML and PHP (and, of course, on the Internet).

Questionnaire and Results on a Single Page

It is certainly possible to execute all the elements of our survey—questionnaire, evaluation, and presentation of results—with a single PHP script. However, as a rule, it is a good idea for the participants not to see the previous results before casting a vote.

Options

You could make the survey more interesting or more informative by offering additional opportunities for input. In our example, there could be a text field for the input of other programming languages or perhaps a text field for comments or for information about the professional background of the participants. However, you must consider that the more you ask, the more superfluous data you will collect. (If someone does not want to answer a required question, you may get a deliberately false response.)

Multiple Selection

In principle, you could construct the questionnaire in such a way that a participant could select more than one programming language. In the HTML form you would use check boxes instead of radio buttons. However, the necessary changes in the design of the database are somewhat more complicated. The table votes must now be able to store in each data record an arbitrary collection of programming languages. It would be easy, though inefficient, to reserve a separate column in the table for each programming language and then store the value 1 or 0 depending on whether or not the language was voted for. If you do not want the votes table to take up any more space in memory than necessary, then you could store a multiple selection as a combination of bits. For this purpose, MySQL offers the data type (SET). But this option would make the program code for the storage and evaluation of the survey results considerably more complicated.

Protection Against Manipulation

In most Internet surveys, the goal is not to obtain information but to promote interest, and thereby obtain a large number of accesses to one's web page. In such cases, there is not much point in protecting the survey against misuse. However, if you wish to provide such protection, you have a number of possibilities:

  • You could place a "cookie" containing a random number on the participant's computer and store this same value in the database. In this way, an attempt to cast additional votes could be easily caught. (The disadvantage is that many Internet users are opposed to cookies and delete all cookies at startup or else prohibit their placement altogether.)

  • You could require a login with e-mail address and password. Only those who have registered may vote. You store information in the database on who has already voted. (The disadvantage is that Internet users tend to have little patience for logins. Not many Internet users would give their e-mail address simply for the privilege of filling out your questionnaire.)

It is true, in general, that you cannot completely prevent such manipulation of a survey. That is, every system of security can be gotten around by someone who puts enough effort into the attempt.

Pointer

There are many complete PHP solutions for questionnaires available, both with and without MySQL support. Before you set about reinventing the wheel, you might want to take a squint at some of the PHP sites on the Internet, for example the following:

http://www.hotscripts.com/PHP/


Part II: Fundamentals

Chapter List

Chapter 4: User Interfaces
Chapter 5: Database Design
Chapter 6: An Introduction to SQL
Chapter 7: SQL Recipes
Chapter 8: InnoDB Tables and Transactions
Chapter 9: Security
Chapter 10: Administration and Server Configuration


Chapter 4: User Interfaces

Overview

The end user should never see MySQL as a program. Instead, a convenient program or several web sites should be used to provide access to the database or assist in the input of new data.

Developers and database administrators, on the other hand, must frequently communicate directly with MySQL, for example, when new databases are created, existing databases enlarged, individual data records added, or queries tested. Such administrative tasks are simplified by user interfaces.

This chapter introduces the most important user interfaces for MySQL. The command program mysql (all operating systems), WinMySQLadmin (Windows), the MySQL Control Center (Windows/Linux), and the HTML interface phpMyAdmin for web administration.

For MySQL there is not merely one user interface, but a host of them. In this chapter we will look at only the most important of these. Here is a brief overview:

  • The command mysql (mysql.exe under Windows) offers the least amount of convenience. It enables the input of text commands and returns results in text format. Nonetheless, this tool cannot be done without. It has the advantages that it is a fixed component of every MySQL installation, is available under all operating systems, makes little demand on hardware, and can be run via telnet or ssh.

  • WinMySQLadmin is provided with the Windows version of MySQL. The program is of use most of all in the setup and configuration of the MySQL server under Windows. However, the program is not a complete user interface.

  • The MySQL Control Center will probably soon become the official interface for MySQL. This program was still in beta version when this book went to press, but it already functions quite reliably. The program currently runs under Unix and Linux. See http://www.mysql.com/products/mycc/

  • phpMyAdmin consists of a collection of PHP scripts. It is used via a web browser, and therefore phpMyAdmin is best suited for the administration of a MySQL server on a remote computer. See ftp://phpmyadmin.sourceforge.net/

All of the above programs are available without charge. But do check the licensing agreement.

Brief Description of Some Other Programs

  • MysqlTool is Perl's answer to phpMyAdmin. It is a collection of Perl CGI scripts that cover all the principal administrative tasks (generate and edit databases and tables, view and edit data, execute queries, set MySQL access privileges, manage MySQL processes, etc.). See http://www.dajoba.com/projects/mysqltool/index.html

  • SQLyog is a compact and very fast Windows program that in addition to the usual administrative tasks, fulfills a host of additional functions: exportation and importation of databases, importation of ODBC data sources, synchronization of two MySQL databases, and so on. If you cannot get on friendly terms with the MySQL Control Center, then this program is perhaps the most attractive alternative. See http://www.webyog.com/sqlyog/index.html

  • DBtools is another Windows program with interesting supplementary functions. Noteworthy are the importation function for Access databases, Excel tables, and ODBC data sources; the PHP script generator; and the support for integrity rules (foreign key constraints) for InnoDB tables. However, using DBtools is less intuitive than is the case for the other programs mentioned here. See http://www.dbtools.com.br/EN/

  • MySQL Navigator is a compact and simple administration program that can be run under both Windows and Linux. See http://sql.kldp.org/mysql/

    Tip

    Download links to many additional user interfaces (including commercial programs) can be found at my web site: http://www.kofler.cc/mysql/


mysql

In the previous chapter you became acquainted with mysql as a simple command interpreter, with which you can execute SQL commands and view the resulting tables in text mode. This section gives some further tips for using mysql with MySQL. Please note that mysql is sometimes called "Monitor," in part to avoid confusion among MySQL in general, the MySQL server, and the program mysql.

Pointer

mysql can also be used for a variety of administrative tasks in batch mode, such as for saving tables in ASCII, HTML, or XML files for saving or restoring previously backed-up databases. These possibilities for using mysql are presented in Chapter 10, where it is described how frequently used mysql options can be stored in configuration files.

For more complex management tasks (backups, ASCII import and export, repairing table files, etc.) there are additional command tools provided with MySQL, such as mysqladmin and myisamchk (see also Chapter 10). A reference to all the options and functions of the command-oriented administration tools can be found in Chapter 19.

Executing mysql and Other MySQL Commands

Unix/Linux: On Unix/Linux systems you simply specify mysql in a console window and press Return. The programs are normally installed in the directory /usr/bin,so that it is unnecessary to provide the name of the directory.

Windows: mysql and most other MySQL commands are generally stored in the directory C:\mysql\bin. Furthermore, C:\mysql\scripts contains several Perl script files (not all of which, however, can be used under Windows). Both directories are normally not a part of the PATH system variable, which contains all directories with executable programs. Therefore, there are the following possibilities for launching the programs:

  • The simplest and most convenient approach is a double click in Windows Explorer. A disadvantage of this procedure is that no options can be passed to the program, yet in most cases it is necessary at the very least to provide the user name and password.

  • Via START |RUN you can launch the program in a small dialog window. There you can also provide options. (You can most easily copy the command name together with all directories with Explorer's Drag&Drop in the start window.) This method is acceptable in isolated instances, but if the command is to be used frequently, this is much too inconvenient.

  • You can open a command window (START |PROGRAMS |ACCESSORIES | COMMAND PROMPT) and there give the name of the program and the complete path. Again, you can save some typing and use Explorer with Drag&Drop to copy the name into the window. Alternatively, you can first use CD to move into the directory C:\mysql\bin.

  • In the long run, it will prove most convenient to extend the system variable PATH to the MySQL tools. To do this execute START |SETTINGS |SYSTEM, click in the dialog sheet ADVANCED, select the system variable PATH with a double click, and enlarge the list of directories to include the MySQL bin directory and the MySQL script directory. Directories must be separated by semicolons. (Figure 4-1 shows the setting of the system variables in Windows 2000.)

    From now on, in every command window (regardless of the current directory or drive), input of the MySQL command name is sufficient to execute it.

    Click To expand
    Figure 4-1: Setting the Windows system variables

Using MySQL Interactively

The program mysql is launched as usual with the options -u and -p. (One can do without the options if the login name is the same as the MySQL user name and MySQL has not been password protected and so a password is not required.)

Now SQL commands can be input. The commands can extend over several lines and must be terminated with a semicolon:

> mysql -u root -p
Enter password: xxx
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 248 to server version: 3.23.24-beta
Type 'help;' or '\h' for help. Type '\c' to clear the buffer
mysql> USE books;
Database changed
mysql> SELECT * FROM title;
titleID    title                                       publisherID    year
      1    Linux, 5th ed.                                        1    2000
      2    Definitive Guide to Excel VBA                         2    2000
      3    Client/Server Survival Guide                          1    1997
      4    Web Application Development with PHP 4.0              3    2000
      6    test                                                  1    2000
      7    MySQL                                                 1    2001
      9    MySQL & mSQL                                          4    1999
7 rows in set (0.01 sec)
Caution

You can do a great deal of damage with mysql. For example, DELETE FROM tablename deletes all data records from the table tablename without so much as a "by your leave." And what is more, there is no way of undoing the damage. If you would like to avoid such fatal errors, you should use mysql with the option --i-am-a-dummy. Then mysql permits the commands UPDATE and DELETE only if they are protected with the key word WHERE or LIMIT. There aresome additional security limitations.

In addition to the SQL commands, you can use some additional commands offered by mysql (for example, source filename). These commands do not need to be terminated with a semicolon. As with SQL commands, these are case-insensitive. Each of the commands exists in a two-character short form, such as \h for help. In the short form the commands can also be given at the end of a line; the only place they must be written out in full is at the beginning of a new line. Table 4-1 gives only the most important commands. A list of all the commands can be found in Chapter 19.

Simplifying Input

Terminating Input: It often occurs, particularly with multiline commands, that you note an error in a line above the one that you are typing and as a result wish to terminate the entire input (so that you can start all over again). To accomplish this, regardless of the cursor position, simply input \c and press Return. Note, however, that \c does not have this effect within a character string enclosed in single or double quotation marks.

History Function: mysql remembers the last commands that were input (even after program termination!). These commands can be recalled with the cursor keys á and ¯.

Table 4-1: mysql commands in interactive mode
\c | clear breaks off input of a command.
\h | help displays the list of commands.
\q | exit or quit terminates mysql; under Unix/Linux this can also be done with Ctrl + D.
\s | status displays status information of the MySQL server.
\T [f] |
 | tee [filename] logs all input and output in the specified file.
\t | notee ends tee; the logging can be resumed at any time with tee or \T; note that the file name must be given again.
\u db | use database makes the specified database the default.
\# | rehash creates an internal list of all mysql commands, the most important SQL key words, and all names of tables and columns of the current database; then it suffices to input only the initial letters; with Tab the initial letters are expanded into the complete key word. Unfortunately, this functions only under Unix/Linux, where rehash is executed automatically.
\. fn | source filename executes the SQL commands contained in the specified file; the commands must be separated by semicolons.

Tips for Using mysql Under Unix/Linux

Scroll Region: Under Unix/Linux, mysql is usually executed in a shell window. Every such program (xterm (Figure 4-2), kvt, nxterm, gnome-terminal, etc.) offers the possibility of setting the number of lines that are kept in temporary storage. If you provide a large enough value, then you can use a scroll bar to examine previously input commands and copy them with the mouse to the clipboard.

Click To expand
Figure 4-2: Using the MySQL monitor under Linux in an xterm window

Keyboard Shortcuts: Under most operating systems, mysql relies on the readline library. Therefore, the usual suspects in the lineup of Unix/Linux keyboard shortcuts are available for editing input (for example, Ctrl + K for deleting a line from the cursor location to the end, Ctrl + Y for restoring the most recently deleted text). Most of the keyboard shortcuts correspond to those under the editor Emacs. Furthermore, names of tables and columns can be automatically completed with Tab, after the initial letters have been input (see also rehash).

Private Configuration File: Frequently used settings or options (such as user name and password) can be stored under Unix/Linux in a user-specific configuration file with the name ~/.my.cnf. Options related to all client tools are placed in the group [client], while those relating specifically to mysql go into the group [mysql]. The following lines provide an example. (Further details on creating configuration files can be found in Chapter 19.)

# Options for all MySQL tools
[client]
user=username
password=xxx
# Options for mysql
[mysql]
database=mydatabase

Since the file contains a password in plain text, it should be protected from prying eyes:

user$ chmod 600 ~/.my.cnf

Tips for Using mysql Under Windows

mysql is executed in a command window. Many of the particulars of this program are configurable. You can access the configuration dialog by clicking on the title bar with the right mouse button (see Figure 4-3).

  • Color: In the command window are usually displayed white characters on a black background. That is not particularly easy on the eyes. If your eyes would prefer a white background and black characters, then consult the COLORS dialog sheet to change the setting.

  • Window Size: There is not much room in an 80 × 25 character display for representing the results of SELECT. The dialog sheet LAYOUT comes to your aid here, allowing you to change the window size.

  • Scroll Region: A larger window does nothing to increase the number of lines displayed. This setting can be changed by enlarging the SCREEN BUFFER SIZE in the dialog sheet LAYOUT. This sets the virtual window size. In the window, a segment of the virtual window is visible, and you can use the scroll bar to see previous commands and results.

  • Clipboard: The command window offers the not-well-known possibility of copying and pasting text to and from the clipboard. To copy, select a rectangular region with the mouse and simply hit Return. To paste, click anywhere in the window with the right mouse button. For these shortcuts to function you must have activated the option QUICKEDIT MODE in the OPTIONS dialog sheet.

    Click To expand
    Figure 4-3: Configuration of the MySQL monitor under Windows

When you close the configuration dialog, you are asked whether the settings should hold only for the current window or whether they should be saved, with the result that the shortcut (link) to the input window will be changed. Choosing the second variant will ensure that in the future, the command window will have the properties described here.

Caution

In MySQL databases, the character set latin1 is used by default. In the command window, however, what is used is an older (DOS-compatible) character set. For this reason, various special characters (such as äåéèöüß) are incorrectly displayed in the command window. What is worse, the DOS character set is also used in data records that are stored using an INSERT or UPDATE command in the command window. If you then look at those data records with another program (for example, with phpMyAdmin), the special characters are seen to be incorrect.

The solution is not to input data records containing such special characters in the command window with mysql. Make such inputs either with another program or with mysql under another operating system (Unix/Linux). If this is impossible, write the input command with a text editor (such as Notepad) and paste it into the command window via the clipboard.

Accessing an External Server with mysql

If on launching mysql you provide the option --host computername, then the program establishes a connection to the MySQL server running on computername. However, for this to function, certain conditions must be met:

  • computername must be reachable over the network. This can be checked, for example, with the command ping computername.

  • The TCP/IP port 3306 must be usable. If there is a firewall between your computer and the computer on which the MySQL server is running, it can happen that the firewall blocks communication over this port.

  • The MySQL server must be so configured that it allows connections from your computer. This is not the case by default! Information on the correct configuration of access privileges is discussed in Chapter 9.

    Remark

    If you execute mysql under Unix/Linux, then the option -h has meaning when you establish a connection to a MySQL server on the local computer. If you specify -h localhost (or ignore the option entirely), then the connection is established over a so-called socket file. On the other hand, if you specify the name of the local computer or its IP number (-h mycomputer), then the connection is established via TCP/IP. Different access settings can hold for these two variants. See Chapter 9.

Terminating mysql

You can terminate mysql with exit or quit or \q or Ctrl + D (the last of these only under Unix/Linux). Under Windows, you can also simply close the command window.


WinMySQLadmin (Windows)

WinMySQLadmin was originally conceived as a complete Windows interface for MySQL. However, the development of this program long ago came to a halt, and its development has been terminated. (In the meanwhile, a number of newer interfaces for Windows have been developed, which function considerably better.)

The program is included with the Windows version of MySQL and is automatically installed. Even though it does not offer a full-fledged user interface, it is nevertheless a useful tool. Namely, it helps in starting, stopping, and configuring the MySQL server.

Remark

If you wish to startor stop the MySQL server with WinMySQLadmin under Windows NT/2000, you must log into Windows as Administrator or as a user with Administrator privileges.

After the first launch (which is done manually), the program registers itself in START |PROGRAMS |STARTUP and in future will be launched automatically at startup. WinMySQLadmin, for its part, automatically launches the MySQL server if it is not already running. (This function is especially practical under Windows 9x/ME, where the MySQL server cannot be executed as a Windows service.)

The program is usually visible as a small icon in the Windows task bar. It uses a stoplight to indicate the state of the MySQL server (see Figure 4-4). A green lamp indicates that the server is running, while red indicates that it is not.

Figure 4-4: WinMySQLadmin as an icon

Displaying WinMySQLadmin

After it has been launched, WinMySQLadmin shrinks to a stoplight icon. If you click on this icon with the right mouse button, you can display WinMySQLadmin with SHOW ME. The button HIDE ME in the dialog sheet Environment shrinks the window again into an icon. In this dialog sheet you can also summon the on-line help for WinMySQLadmin, which, unfortunately, is not much help at all.

Starting and Stopping the MySQL Server as a Service Under Windows NT/2000/XP

When WinMySQLadmin is executed under Windows NT/2000/XP, it automatically registers the MySQL server as an operating system service and then launches this service. Thereafter, you can use WinMySQLadmin to stop the service and to start it again. You can also unregister the service with WinMySQLadmin, but at the next startup of WinMySQLadmin, the service will be automatically registered again.

MySQL Configuration

At first launch, WinMySQLadmin creates the file Windows\my.ini if such a file does not yet exist. This file can be viewed and edited in the dialog sheet MY.INI SETUP.

Sad to tell, the entire user interface in the dialog sheet (see Figure 4-5) is a bit complicated and confusing. The following description explains the effect of the individual dialog elements and how they are related to one another:

  • BASE DIR: In this text field, the MySQL installation directory can be set (for example, C:\mysql or Program\mysql). You can then find in this directory the directories data,with the MySQL databases, and bin, with the MySQL programs and commands. A change in this directory is generally not necessary.

    The directory set here is used to determine the file names for the MySQL server (bin\mysqld.exe is simply added to the directory).

  • MYSQLD FILE: With these options one can set which of the MySQL server variants should be launched. Normally, you should use mysqld-opt.exe under Windows 9x/ME, and mysqld-nt.exe under Windows NT/2000/XP. Details on the different versions can be found in Chapter 2.

    Which of these variants you want to use should be specified in MySQLadmin, in the dialog sheet MY.INI SETUP. Under windows 9x/ME, you then stop MySQL and restart the server. Under Windows NT/2000/XP, the process is somewhat more complex: You stop the server, deinstall the service (REMOVE SERVICE), reinstall the service, and then restart the server. All steps are carried within a context menu, obtained with a right mouse click.

  • PICK-UP AND EDIT MY.INI: With this button, Windows\my.ini is read anew and displayed in the dialog. Any changes made in WinMySQLadmin will be lost.

  • SAVE MODIFICATIONS: With this button, first the settings BASE DIR and MYSQLD FILE are evaluated and saved in the option Server of the group [WinMySQLadmin]. Then Windows\my.ini is saved.

  • CREATE SHORT CUT: With this button, the entry PROGRAMS | STARTUP |WINMYSQLADMIN is generated in the Windows start menu (if it does not yet exist).

    Caution

    For specifying path names in the configuration file, the forward slash / must be used, not the backslash \, as is usual under Windows.
    Remark

    Please note that there can be two additional configuration files for the MySQL server: C:\my.cnf and DATADIR\my.cnf. If thereare several configuration files existing simultaneously, they are read in the following order:

    C:\my.cnf ® Windows\my.ini ® DATADIR\my.cnf ( only mysqld).

    Settings in the last file to be read have priority. Using configuration files is discussed in detail in Chapter 19.
Click To expand
Figure 4-5: Setting my.ini in WinMySQLadmin

Displaying Status and Database Structure Information

WinMySQLadmin provides much information in the dialog sheets ENVIRONMENT, SERVER, VARIABLES, and PROCESS about the state of the MySQL server: status information, the contents of the status and system variables, and a list of all running MySQL processes. Furthermore, the DATABASE dialog sheet gives a quick overview of the databases, rows, and columns managed by MySQL. Unfortunately, the database schema cannot be altered; nor can the content of tables be examined.

However, WinMySQLadmin can display this status information only if the MySQL server makes this program available. WinMySQLadmin must declare itself to the MySQL server just like any other client. The program uses the user name and password combination stored in Windows\my.ini. These two entries are set during the initial launch of WinMySQLadmin:

[WinMySQLadmin]
user=username
password=xxx
. . .

If these two lines are missing in my.ini, the program will attempt to register with the MySQL server without a password as odbc@localhost. This will work only if the access rules of the MySQL server allow registration without a password.

Caution

Since by default, Windows\my.ini can be read by everyWindows user, you should not put the password for root access to the MySQL server into this file. That would be a tremendous security risk.

Examining Error Logs and Creating Bug Reports

In the dialog sheet ERRFILE, the (most recent) lines of the file hostname.err should be displayed. However, this no longer functions in the most recent version of WinMySQLadmin.

The dialog sheet REPORT is provided for the purpose of preparing an error report, which can then be sent to the MySQL company or to a mailing list. The button CREATE

REPORT collects all available status information about the MySQL server and places it in a text field. This text can be copied and inserted into an e-mail or word processing program.


MySQL Control Center (Windows, Linux)

The MySQL Control Center, or MyCC for short, is currently the most promising user interface, brought to you directly from the MySQL company. The program uses a general public license (GPL). Commercial use is without charge if one has a commercial license for MySQL.

The Control Center is available in the current version (0.8.10 beta) as pre-compiled code for Windows and Linux (glibc 2.2 and 2.3) at the following address: http://www.mysql.com/products/mysqlcc/index.html.

The Windows installation is carried out with a setup program. Under Linux, the tar.gz archive must be extracted with tar to the desired directory. Then a link to the file mysqlcc must be established from an arbitrary bin directory:

root# cd /usr/local
root# tar xzf mysqlcc-n.tar.gz
root# cd /usr/X11R6/bin
root# ln -s /usr/local/mysqlcc-n/mysqlcc mysqlcc

Those who use other Unix versions must compile the Control Center themselves. The program is based internally on the Qt library. (It is to be expected that the program will exist in several binary versions once it reaches the status stable. Even a version for MacOSXis planned.)

Adding a Server

The Control Center allows for the administration of several MySQL servers (including those not running on the local computer). However, first, for each server, with FILE |NEW you must set up a dialog with all the connection parameters (Figure 4-6). If the server is running on the same computer as the Control Center, the correct setting for HOST NAME is simply localhost. As long as MySQL is not password secured, use root as user name and do not specify a password.

Warning

If the MySQL server is already secured (see Chapter 9), the temptation is great to specify the password in the password dialog. This is, however, a great security risk, since it is stored in plain text in the configuration files (under Linux in the directory ~/.mysqlcc/, under Windows in the directory Documents and settings\usernamme\.mysqlcc\).
Click To expand
Figure 4-6: Setting up a new server

Using the Control Center

Using the MySQL Control Center is quite easy. In the window CONSOLE MANAGER a listbox in the form of a tree is displayed on the left, in which you can select from among all MySQL servers, their databases, and their tables (see Figure 4-7). A double click on a table displays the data contained within. All other operations are performed through either a context menu or the ACTION menu.

Click To expand
Figure 4-7: Database administration with the MySQL Control Center

Managing Databases and Tables

Using the context menus in the listbox, you can perform such tasks as creating new databases and tables, checking a table's integrity (CHECK TABLE), or displaying the CREATE TABLE command. In designing tables, a dialog is available to assist you (see Figure 4-7, lower right), in which you can name the individual columns, set their data type and additional attributes, define indexes, and change table properties (e.g., the type of table). With the same dialog you can also change existing tables.

In the windows that show the content of a table you can also alter existing data, insert new data records, and delete existing data records.

Caution

The program deletes all records that are selected or in which the cursor is located. If you select entire rows by clicking on the row numbers, then the cursor remains where it was (by default on the first row). This leads to the possibility of deleting one data record too many unintentionally. The solution is to select the records to be deleted by selecting a range of cells inside the table with the mouse, not entire rows.

BLOB fields with binary data can also be displayed with a double click in JPEG, PNG, and GIF format. (This functions only if bitmap data are available.)

Executing SQL Commands

In every table window a click on the button SQL makes a range of cells visible in which the underlying SELECT command is displayed. Here, you can input new SQL commands (where several commands are to be separated by a semicolon). To execute them, press Ctrl+E or click the button with the exclamation mark. In the SQL code, key words and character strings are shown in color, which simplifies the input process. (See Figure 4-8.)

Click To expand
Figure 4-8: Executing SQL commands
Tip

Unfortunately, the Control Center currently offers no import or export commands. However, you can always store the results of an SQL query or the contents of a table as a text file or load a text file with SQL commands. For this, you use the FILE command of the table window.

Server Administration

In the listbox of the CONSOLE MANAGER, under the server's databases, there are two additional entries: SERVER ADMINISTRATION and USER ADMINISTRATION. A double click on SERVER ADMINISTRATION leads to a window that displays all MySQL processes, variables, and status information. Furthermore, via a context menu you can execute various FLUSH commands and even shut down the server with SHUTDOWN. However, the program offers no way of restarting the server; for that, you have to use WinMySQLadmin under Windows, and execute /etc/init.d/mysql[d] start under Unix/Linux.

User Administration

The entry USER ADMINISTRATION leads to a list of all users who can access the MySQL server. A double click on a user allows the alteration of that user's privileges. The dialog for this purpose is easy to use (see Figure 4-9), and of course, all of this assumes that you know and understand the access system for MySQL. (A detailed description appears in Chapter 9.)

Click To expand
Figure 4-9: User administration with the MySQL Control Center.


phpMyAdmin (HTML)

phpMyAdmin consists of a number of PHP scripts that enable nearly all of the administration of MySQL. Thanks to the HTML interface the administration can be carried out with almost any web browser, independent of operating system, locally as well as over a network.

Overview of the User Interface

The user interface of phpMyAdmin consists principally of three HTML pages (based on various PHP scripts). If you work with phpMyAdmin, you will, of course, stumble on many pages, but the three discussed here (and we shall continually refer to them in this section) represent the starting point for all navigation with phpMyAdmin.

  • Start Page (Figure 4-10): The initial page index.php3 enables the selection of desired MySQL servers (if phpMyAdmin was set up for the administration of several servers), the creation of a new database, and the execution of various administrative tasks related to the server. (Depending on the access rights phpMyAdmin has with the MySQL server, many links will be missing. For example, if you do not have the right to generate new databases, then phpMyAdmin does not even offer this command.) Using the listbox in the left area of the start page, you can select a database for further processing.

    Click To expand
    Figure 4-10: Start page of phpMyAdmin

  • Database View (Figure 4-11): The database view provides an overview of all tables of the database and shows a collection of various functions for processing the entire database (generate and delete tables, execute SQL commands, export the database, etc.).

    Click To expand
    Figure 4-11: Database view of phpMyAdmin

  • Table View (Figure 4-12): In table view, you see all the columns of the database with their properties. Here you can insert columns, edit or delete, and execute various administrative functions for individual tables (rename tables, copy, export, import ASCII files, etc.).

    Click To expand
    Figure 4-12: php table view

Pointer

You can find extensive on-line help via the start page of phpMyAdmin by following the link PHPMYADMIN DOCUMENTATION.

Installation and Configuration

Installation Variants

There are several possibilities for installing and using phpMyAdmin:

  • In the simplest case, both phpMyAdmin and MySQL run on the same local computer (that is, Apache, PHP, MySQL, and the scripts for phpMyAdmin are all locally installed). This installation variant has already been described in Chapter 2.

  • It is not much more complicated to install phpMyAdmin locally and thereby maintain a MySQL server that is running on another computer. The only difference is that config.inc.php must be set so that phpMyAdmin creates a connection to the external MySQL server. However, often, the problem arises that MySQL must be configured on the external computer in such a way that external administration is permitted. But for security reasons, ISPs in particular usually permit only local access to the server (that is, by localhost).

  • For the administration of a MySQL server located at an ISP, it is worthwhile to install phpMyAdmin there. As a rule, only the script files of phpMyAdmin need to be transported into a directory of the web server. All other conditions (Apache, PHP) are automatically fulfilled by most ISPs. Even access to MySQL is now not a problem, since phpMyAdmin is now, from the point of view of MySQL, being executed locally.

    Caution

    It happens frequently that phpMyAdmin is installed in such a way that it is accessible to all users over a local network or even over the Internet who know or can guess the network address of the start page. To avoid giving complete MySQL access to just anyone, the access and phpMyAdmin script files must be password protected. Usually, a .htaccess file is used. Introductory information on dealing with such files can be found in Chapter 2. Further information can be found in any half-decent book on Apache.

Configuration of MySQL Access

The complete configuration of phpMyAdmin is accomplished in the file config.inc.php.In the simplest case, it suffices to set the following variables:

# configuration file config.inc.php
. . .
// must contain the web address of the phpMyAdmin directory
$cfg['PmaAbsoluteUri'] =
'http://computer.name/phpmyadmin directory/';
. . .
$cfg['Servers'][$i]['host']      = 'computer.name'; // or 'localhost'
. . .
$cfg['Servers'][$i]['auth_type'] = 'config';       // must be 'config'
$cfg['Servers'][$i]['user']      = 'root';         // MySQL user
$cfg['Servers'][$i]['password']  = 'xxxx';        // MySQL password
...

If the MySQL server is running on the same computer as the web server that is executing the phpMyAdmin script files with PHP, then you give localhost as the computer name. Optionally, with $cfgServers[1]['port'] the TCP/IP port over which the link is established can be specified. That is necessary only if the default port 3306 is not the one to be used.

Caution

The file config.inc.php contains the user name and password in plain text. That is always a security risk. Make sure that unauthorized access to this file, such as via anonymous FTP, is not possible.
Remark

This section describes phpMyAdmin 2.4. If you update from an earlier version, you should not take the configuration file config.inc.php from the older installation (copy), but create a new file, because the syntax of this file has changed somewhat.
Remark

The extent of the possibilities for administration with phpMyAdmin depends, of course, on the MySQL access rights possessed by the MySQL user whose name phpMyAdmin has used at login (that is, $cfgServers[1] ['user']= ... ). Therefore, phpMyAdmin is not responsible for access protection, but rather the MySQL access system with the privilege database mysql, which is described fully in Chapter 9.

Conversely, we see that if only some of the options for phpMyAdmin described below work for you, then it is not that phpMyAdmin is flawed, but that you have insufficient access rights to MySQL. In particular, if you use phpMyAdmin to manage a database located at an ISP, you will generally have access only to your own database. (Otherwise, your ISP would be guilty of a serious breach of administrative etiquette.)

Servicing Several MySQL Servers with phpMyAdmin

Host, user, and password entries for multiple MySQL servers are possible in config.inc.php. If you make such entries, then on the start page of MySQL there appears a listbox with the names of the computers whose MySQL servers can be managed. Selecting one of these names establishes a link to that server.

The variable $cfg['ServerDefault'] determines the server that phpMyAdmin uses to establish a link at startup. If the variable is set to 0, then at startup phpMyAdmin initially sets no link, but merely presents a listbox with the selection of servers.

If you do not wish to have simply the computer name (array variable host) displayed in the listbox, then you can use $cfg['Servers'][$i]['verbose'] to display a different character string.

Additional Configuration Possibilities

In addition to the MySQL access data in config.inc.php, you can also set various options. The following list describes only the most important of these, together with their associated variables:

  • $cfg['Confirm'] = true/false: This variable specifies whether a confirmation query should be displayed before data are deleted (default: true).

  • $cfg['MaxRows'] = n: Specifies how many data records should be displayed per page as the result of a query (default is 30).

  • $cfg['ExecTimeLimit'] = n: Specifies how long the execution of the PHP code will be allowed to run (default is 300 seconds). The time is relevant primarily for time-intensive import/export commands.

  • $cfg['ShowBlob']=true/false: Specifies whether the contents of fields of type xxxBLOB are to be displayed (default: false).

  • $cfg['ProtectBinary'] = false/'blob'/'all': Specifies whether the contents of BLOB fields are to be write protected. By default ('blob'), BLOBs cannot be changed. The prohibition against change to any table content (regardless of data type) is effected by 'all', while 'false' permits everything to be changed.

  • $cfg['LeftWidth']=n: Specifies the width of the left column of the main window (by default, 150 pixels). In this column are displayed the listbox for database selection and the list of tables in the current database.

http and Cookie Authentication

The configuration described in the previous section with $cfg['Servers'][$i]['auth_type'] = 'config' is simple, but it is inflexible if various users wish to manage their own databases, and insecure because user names and passwords appear in plain text in config.inc.php. If you as an Internet service provider wish to allow many users uniform access to phpMyAdmin, you should consider the two configuration variants described here.

Assumptions

  • This section assumes that you are familiar with MySQL and in particular, with its security system (see Chapter 9). If that is not yet the case, you should skip this section and stick with the standard configuration described above (which is sufficient for most purposes).

  • You must have administrator privileges for the MySQL server. That is not the case for installing phpMyAdmin on an ISP's computer.

  • It is assumed in http authentication that the PHP interpreter is executed as an Apache module (and not as a CGI program). This is generally the case with current versions of PHP (even under Windows).

  • cookie authentication is designed for the case in which http authentication is not possible (for example, when the Microsoft Internet Information Server (IIS) is used as the web server). This form of authentication assumes that the user permits cookies from phpMyAdmin and that the PHP interpreter and the user's browser are in agreement on the management of cookies. The phpMyAdmin documentation discusses problems with Internet Explorer (through version 6) and with PHP 4.1.1.

Creating a MySQL User for phpMyAdmin

Before http or cookie authentication can be used, a MySQL user must be created that can read mysql.user, mysql.db, and mysql.tables_priv (that is, it must be provided with the SELECT privilege).

The MySQL administrator can create such a user with the following SQL commands. The name of this user is irrelevant. For the following example, the name pmaUser will be used:

GRANT USAGE ON mysql.* TO pmaUser@localhost
  IDENTIFIED BY 'xxxx'
GRANT SELECT (Host, User, Select_priv, Insert_priv, Update_priv,
    Delete_priv, Create_priv, Drop_priv, Reload_priv, Shutdown_priv,
    Process_priv, File_priv, Grant_priv, References_priv, Index_priv,
    Alter_priv)
  ON mysql.user TO pmaUser@localhost;
GRANT SELECT ON mysql.db TO pmaUser@localhost
GRANT SELECT (Host, Db, User, Table_name, Table_priv, Column_priv)
  ON mysql.tables_priv TO pmaUser@localhost
http authentication

In order to be able to use phpMyAdmin http authentication, make the changes shown in boldface to the file config.inc.php:

# in config.inc.php
$cfg['Servers'][$i]['controluser'] = 'pmaUser'; // Read-only-User
$cfg['Servers'][$i]['controlpass'] = 'xxx';     // password
$cfg['Servers'][$i]['auth_type']   = 'http';    // http authentication
$cfg['Servers'][$i]['user']        = '';       // empty!
$cfg['Servers'][$i]['password']    = '';      // empty!

Now, every time that you want to use phpMyAdmin, the web browser's login dialog appears (see Figure 4-13), in which you must give a MySQL user name and associated password. For further work in phpMyAdmin, you then have the privileges of this MySQL user.

Click To expand
Figure 4-13: Login dialog for http authentication

As always, config.inc.php contains a user name and password (pmaUser and xxx). But now the security risk is much less, because this MySQL user can read only a few tables on the MySQL server and cannot change any of them.

Remark

Due to an error in phpMyAdmin 2.4, it is impossible to log in twice consecutively with the same user name and password. To bypass this error, you have to quit the web browser and restart.
cookie Authentication

For cookie authentication, you must change only the auth_type line in config.inc.php:

# in config.inc.php
$cfg['Servers'][$i]['controluser'] = 'pmaUser'; // Read-only-User
$cfg['Servers'][$i]['controlpass'] = 'xxx';     // password
$cfg['Servers'][$i]['auth_type']   = 'cookie';  // cookie authentication
$cfg['Servers'][$i]['user']        = '';       // empty!
$cfg['Servers'][$i]['password']    = '';      // empty!

Login errors are not shown within the phpMyAdmin start page (see Figure 4-14).

Click To expand
Figure 4-14: Login dialog for cookie authentication

Creating and Editing Databases

If your desires run in the direction of creating a new database, phpMyAdmin can save you the necessity of having to deal with many complex CREATE TABLE commands.

Creating a Database

The creation of a new database begins on the start page of phpMyAdmin, in which you provide the name of the new database. This database then becomes the active database. (If you wish to insert a table into a preexisting database, then click on that database in the list of databases.)

Creating a Table

On the table page you will find, among other things, CREATE NEW TABLE. There you give the desired name of the table and the number of columns (when in doubt, just give a number that is too large. You can always leave the extra columns empty). OK leads to the next form (see Figure 4-15), in which you can conveniently set the names and properties of the columns.

Click To expand
Figure 4-15: Design of the table titles in phpMyAdmin

The use of phpMyAdmin requires a bit of knowledge about the possible and necessary settings for the various data types. For example, you will get slapped with an error message if you attempt to declare a VARCHAR column without providing a length, or a TEXT column with a length (neither variant is provided for in MySQL). You may then, of course, correct your error.

Changing the Table Design

In the phpMyAdmin table view (Figure 4-16) you can equip each column with an index by a simple click of the mouse. It is just as simple to change the column attributes, such as the maximal length of character strings (link CHANGE). You can delete existing columns and insert new columns at any position (ADD NEW FIELD).

Click To expand
Figure 4-16: Table view with many options for changing the design

Inserting, Displaying, and Editing Table Data

Inserting Data into a Table

In the dialog sheet INSERT (Figure 4-17) you can insert individual data records into a table.

Click To expand
Figure 4-17: Adding a data record

This is a convenient way of proceeding for a few test records. However, only one data record can be edited (not several simultaneously), so this procedure is rather cumbersome for larger data sets.

  • Functions: It is often the case in the input of a data record that an SQL function is to be inserted, such as PASSWORD('xxx') to store xxx in encrypted form. For this, phpMyAdmin offers several frequently used functions in the column FUNCTION, including ASCII, CURTIME, NOW.Figure 4-18 demonstrates the use of such functions.

    Click To expand
    Figure 4-18: Adding a password

    Tip

    It is relatively tedious to input several data records with phpMyAdmin. In many cases, it is more efficient to input the records and then import them with INSERT TEXTFILE INTO TABLE on the dialog sheet STRUCTURE.

Displaying Table Contents

In both database view (see Figure 4-11) and table view (see Figure 4-16) the link BROWSE leads to a display of the contents of the table. For this, the command SELECT * FROM table LIMIT 0, 30 is executed. You may then move backward and forward through the table.

Tip

If you would like to sort a table according to a different criterion, simply click in the relevant column.
Editing Existing Data

If you click on the link EDIT shown in Figure 4-19, you will land on a new page, in which the data record can be edited. This page looks as depicted in Figure 4-17, where in the column VALUE the previous contents of the record are displayed, where they can be easily edited.

Remark

A change in data is possible only if the table has a column identified as PRIMARY KEY (usually with id values that identify each record).
Click To expand
Figure 4-19: Page-by-page representation of data records

Server Administration

Executing SQL Commands

In the dialog sheet SQL, in table view (see Figure 4-11) you can input SQL commands. With OK, the command is then executed for the current database. The SQL command does not have to be terminated with a semicolon, as in mysql. However, a semicolon is necessary if several commands are to be executed simultaneously.

If the command involves SELECT, then the results are displayed page by page, like the contents of a table. However, the links EDIT and DELETE function only if the query results come from a single table (no JOINs from several tables) and the PRIMARY KEY column of the table is a part of the query results.

Instead of inputting SQL commands, you can input instead the name of a file containing SQL commands. Then phpMyAdmin loads this file and executes the commands contained therein. This procedure can also be used to load a previously executed phpMyAdmin database backup.

Remark

phpMyAdmin also offers a query editor. However, the interface is rather a disappointment, and it is difficult to use. Without a good fundamental knowledge of SQL it is practically impossible to link two tables. As a rule, it is easier just to type in the SQL code.
Executing Backups, Saving the Database Structure

In both database and table views, one has available the dialog sheet EXPORT (see Figure 4-20). Here you can transform an entire database (or just the design or the data) or individual tables into a text file.

There are several options:

  • STRUCTURE ONLY: phpMyAdmin generates SQL commands to generate new empty tables.

  • STRUCTURE AND DATA: phpMyAdmin generates SQL commands to generate new tables and fill them with data (INSERT commands).

  • DATA ONLY: phpMyAdmin generates only INSERT commands for restoring data.

  • LATEX: phpMyAdmin generates a text file for representing data as a LATEX table. (LATEX is program available under Unix/Linux and other operating systems for typesetting scientific texts. This book was typeset in LATEX.)

  • CSV DATA: phpMyAdmin generates a text file that can be imported into Excel or another program. (CSV stands for comma-separated values.)

  • XML: phpMyAdmin generates a text file in XML format.

  • WITH DROP TABLE: Before every CREATE TABLE command a DROP TABLE command is inserted. If you later use the resulting file to recreate the table, then this file will first be deleted.

  • COMPLETE INSERTS: For every INSERT command, phpMyAdmin also specifies the column name, for example, in the form INSERT INTO author (authorID, author) VALUES ( '1', 'name').Without this option, the INSERT command would be generated in a more compact form: INSERT INTO author VALUES ('1', 'name').

    COMPLETE INSERTS are necessary if the data are to be inserted from one table into another that exhibits additional columns or whose arrangement of columns is different from that in the initial table.

  • EXTENDED INSERTS: phpMyAdmin joins all INSERT commands into a single long command: INSERT INTO author VALUES (1, 'name1'), (2, 'name2') ... .

  • Send: phpMyAdmin does not display the resulting text file, but sends it to the user (as with a download). The data can optionally be compressed to lessen transmission time.

    Click To expand
    Figure 4-20: Executing a database backup

    Remark

    The exportation of very large databases and tables is problematic, because a PHP script's maximal run time is limited. You can attempt to increase this time in config.inc.php ($cfg['ExecTimeLimit'] = n), which does not always help (depending on how PHP itself is configured).

Database Upload

To read in a database (database uploading), the database must first be created. Then in the dialog sheet SQL you use BROWSE to select the name of the SQL file and execute the SQL commands contained therein with OK.

Remark

As with exporting databases, there can be problems in importing large databases or tables. In addition to the time limit, PHP also limits the size of upload files (setting upload_max_filesize = 2M in php.ini). When PHP is executed on your Internet service provider's computer, you have no influence over the file php.ini. (Of course, you can ask your ISP to change certain variables, but experience suggests that you will not meet with an enthusiastic response.) The only solution is usually to break the upload file into smaller units and read them in one at a time.
Importing Text

What do you do when you have a text file with data (not SQL commands) that you would like to import into a table? Here, too, phpMyAdmin comes to your assistance. In the dialog sheet SQL of table view you will find the link INSERT TEXTFILES INTO TABLE. This link leads you to the dialog depicted in Figure 4-21. There you specify the name of the file. Moreover, you can set a number of options that format the text file.

Click To expand
Figure 4-21: Text importation of table data
Pointer

The meaning of the various options is detailed in the description of the command LOAD DATA in Chapter 18. Examples of the application of LOAD DATA, which can also be used for importing with phpMyAdmin, can be found in Chapter 10.

NULL is usually represented in a text file as \N (with uppercase N,not n).
Renaming and Copying Tables, Changing the Table Type

The dialog sheet OPERATIONS in table view contains commands for renaming and copying tables and transferring tables to another database.

If you wish to change the type of a table, you can do this in the dialog sheet OPTIONS.

Commands for deleting indexes or inserting new ones can be found in the dialog sheet STRUCTURE. However, phpMyAdmin 2.4 does not offer the possibility of defining integrity rules (foreign key constraints) for InnoDB tables.

At the bottom of the table view you will find the commands RENAME TABLE TO and COPY TABLE TO. With RENAME you can, as you might expect, give a table a new name. With COPY you make a duplicate table within the same database, where either just the table structure is created or the entire table with its contents. If you wish to copy a table into another database, you specify the new table name in the form databasename.tablename.

Other Administrative Tasks

If you click on HOME at the top of the database tree (left area of the window), you find yourself back at the phpMyAdmin start page. If you have sufficient phpMyAdmin login privileges, this page contains links for carrying out various administrative tasks:

  • SHOW MYSQL RUNTIME INFORMATION displays the status of the MySQL server (corresponds to SHOW STATUS).

  • SHOW MYSQL SYSTEM VARIABLES displays the content of the MySQL variables (corresponds to SHOW VARIABLES).

  • SHOW PROCESSES displays the current process list (corresponds to SHOW PROCESSLIST). If you have sufficient privileges, you can also terminate a process.

  • PRIVILEGES leads to a page that lists all MySQL users. From this page you can add new users, change the privileges of existing users, and delete users. This assumes, however, that you already are familiar with the MySQL access system (see Chapter 9).

  • DATABASES leads to a list of all databases. With ACTIVATE DATABASE STATISTICS you can determine the sizes of all databases. Also practical is the link CHECK PRIVILEGES, which leads to a page that tells which MySQL users have access to a database.

Additional Functions

The program phpMyAdmin recognizes some additional functions that are available only when phpMyAdmin has been configured for their use and can store data in tables especially for such purposes. This section describes how you can use these additional functions. We assume that you are already familiar with MySQL and are capable of creating new databases and tables and executing SQL commands.

Creating a Database for phpMyAdmin

The program phpMyAdmin requires a database whose tables it is permitted to read and edit. If you have installed phpMyAdmin on your website, where the MySQL server is managed by your ISP, you are usually not permitted to create new databases. In this case, you simply use one of your existing databases. The name of this database is specified in the phpMyAdmin configuration file config.inc.php:

#in config.inc.php
$cfg['Servers'][$i]['pmadb'] = 'mydatabase';

If you are permitted to create new databases, then it is simpler to create a new database for storing phpMyAdmin administrative data. This database can have whatever name you choose to bestow on it. In the following, we will eschew our wildest flights of fancy and choose the name that appears in the phpMyAdmin documentation: phpmyadmin.

CREATE DATABASE phpmyadmin
#in config.inc.php
$cfg['Servers'][$i]['pmadb'] = 'phpmyadmin';

The MySQL login under which phpMyAdmin runs must have read and write privileges for this database. If you have set up phpMyAdmin for http or cookie authentication, as discussed previously, the phpMyAdmin controluser must have these privileges. For this, you execute the following command:

GRANT SELECT,INSERT,UPDATE,DELETE ON phpmyadmin.*
TO pmaUser@localhost
Saving SQL Queries (bookmarks)

If you frequently execute SQL commands in phpMyAdmin, it is burdensome to have to input the same command over and over. If you create the table PMA_bookmark, then in the future, you can give each of your SQL commands a name and store it. The table PMA_bookmark is generated by the following command (if you use your own database for storing the phpMyAdmin administrative data, then the command must be called CREATE TABLE mydatabase.PMA_bookmark):

CREATE TABLE phpmyadmin.pma_bookmark (
  id INT NOT NULL AUTO_INCREMENT,
  dbase VARCHAR(255) NOT NULL,
  user VARCHAR(255) NOT NULL,
  label VARCHAR(255) NOT NULL,
  query TEXT NOT NULL,
  PRIMARY KEY (id)
) TYPE=MyISAM COMMENT='Bookmarks'

The name of this database must be specified in config.inc.php.

#in config.inc.php
$cfg['Se vers'][$i]['bookmarktable'] = 'pma_bookmark';

From now on, after the execution of an SQL command, you can store the SQL code. For this purpose, the end of the result page contains a text field for the name of the query, as well as a button called SAVE SQL QUERY. Moreover, you can use the stored SQL code for constructing new SQL commands.

Saving Information About Relations

The program phpMyAdmin is not able to determine which fields are used to link two tables together (not even if integrity rules are defined for InnoDB tables; see Chapter 8). However, if you create the following two tables, then you can specify this information yourself:

CREATE TABLE phpmyadmin.pma_relation (
  master_db     VARCHAR(64) NOT NULL,
  master_table  VARCHAR(64) NOT NULL,
  master_field  VARCHAR(64) NOT NULL,
  foreign_db    VARCHAR(64) NOT NULL,
  foreign_table VARCHAR(64) NOT NULL,
  foreign_field VARCHAR(64) NOT NULL,
  PRIMARY KEY (master_db, master_table, master_field) ,
KEY foreign_field(foreign_db, foreign_table)
) TYPE = MYISAM COMMENT = 'Relation table'
CREATE TABLE phpmyadmin.pma_table_info (
  db_name       VARCHAR(64) NOT NULL,
  table_name    VARCHAR(64) NOT NULL,
  display_field VARCHAR(64) NOT NULL,
  PRIMARY KEY (db_name, table_name)
) TYPE=MyISAM COMMENT='Table information for phpMyAdmin'

The name of this table must, in turn, be supplied to config.inc.php:

# in config.inc.php
$cfg['Servers'][$i]['relation'] = 'pma_relation';
$cfg['Servers'][$i]['table_info'] = 'pma_table_info';
Remark

Even after you have constructed both tables correctly, the STRUCTURE sheet still shows an error message: The additional features for working with linked tables have been deactivated. However, this error message means only that a part of these functions have been deactivated. (These additional functions will be described later.) However, the basic functions should be functioning correctly.

Now in the STRUCTURE sheet of table view, via the link RELATION VIEW you can specify for each foreign key field (foreign key column) which field of another table it refers to. Furthermore, you can specify which columns of the table are displayed when another table refers to it.

This is perhaps most simply explained with an example. The fields publID, catID, and langID in the table titles from the example database mylibrary (see Chapter 5) refer to the tables publishers, categories, and languages. This information was depicted in Figure 4-22. If another table refers to the titles table, than the titleID number should not be displayed, but the book title (field title,Figure 4-22).

Figure 4-22: References by the table titles to other tables in the database mylibrary

The input of this additional information for all tables is, of course, a bit tiresome. In what follows, however, the processing of this database by phpMyAdmin will be significantly easier.

  • When you look at the contents of a table, foreign key fields are executed as links. When you click on a link, the corresonding data record of the linked table is displayed.

  • When you change a data record in a table or input a new one, you no longer have to specify an ID number in the foreign key fields. Instead, listboxes are displayed that enable you to select the linked fields (see Figure 4-23).

  • The QUERY EDITOR now automatically recognizes links between tables and is much simpler to use.

  • The link DATA DICTIONARY in the STRUCTURE page of table view now provides detailed information about the structure of the database and can be easily printed (as documentation about the database).

  • Finally, you can generate a pdf file for documenting the table structure that contains all the relevant information.

    Click To expand
    Figure 4-23: Listboxes simplify the input of foreign key fields.

Saving Column Comments

In version 4.0.n, MySQL allows the storing of comments for every table, but not does offer the option of storing comments on columns. If you wish to do this (for better documentation of your database), you need an additional table:

CREATE TABLE phpmyadmin.pma_column_comments (
  id int(5) UNSIGNED NOT NULL AUTO_INCREMENT,
  db_name VARCHAR(64) NOT NULL,
  table_name VARCHAR(64) NOT NULL,
  column_name VARCHAR(64) NOT NULL,
  comment VARCHAR(255) NOT NULL,
  PRIMARY KEY (id),
  UNIQUE KEY db_name (db_name, table_name, column_name)
) TYPE=MyISAM COMMENT='Comments for Columns'

The associated entry in config.inc.php looks like this:

$cfg['Servers'][$i]['column_comments'] = 'pma_column_comments';

The input of column comments in phpMyAdmin is accomplished in the same dialog as that for relations (link RELATION VIEW in the STRUCTURE page of table view).

Creating pdf Documents

The function introduced in phpMyAdmin 2.3 for creating pdf files to document table design seems a bit unripe. You need two additional tables to execute this function:

CREATE TABLE phpmyadmin.pma_table_coords (
       db_name         VARCHAR(64) NOT NULL,
       table_name      VARCHAR(64) NOT NULL,
       pdf_page_number INT NOT NULL,
       x
     FLOAT UNSIGNED NOT NULL,
       y
     FLOAT UNSIGNED NOT NULL,
       PRIMARY KEY (db_name, table_name, pdf_page_number)
) TYPE=MyISAM COMMENT='Table coordinates for phpMyAdmin PDF output'

CREATE TABLE phpmyadmin.pma_pdf_pages (
       db_name    VARCHAR(64) NOT NULL,
       page_nr    INT(10) UNSIGNED NOT NULL AUTO_INCREMENT,
       page_descr VARCHAR(50) NOT NULL,
       PRIMARY KEY (page_nr),
       KEY (db_name)
) TYPE=MyISAM COMMENT='PDF Relationpages for PMA'

Both tables must be entered in config.inc.php.

$cfg['Servers'][$i]['table_coords'] = 'pma_table_cords';
$cfg['Servers'][$i]['pdf_pages'] = 'pma_pdf_pages';

The creation of a pdf document is accomplished in several steps:

  • The process begins in the STRUCTURE page of database view with the link EDIT PDF PAGES.

  • Now you give a name to the pdf document that is to be created and select it for processing.

  • Now you must select all the tables that are to be documented. (Surprisingly, all tables must be selected individually.)

  • For the structure view in which the links between tables are represented, you must give the start coordinates for each table. The coordinate range is by default from 0 (left) to 300 (right) for the x-coordinate, and from 0 (above) to 200 (below) for the y-coordinate. However, phpMyAdmin automatically adjusts the coordinate range if you place a table outside this region.

    It is rather a bother to arrange tables nicely, and generally, one uses the method of trial and error. (You begin with some arbitrary values, look at the resulting pdf page, make an adjustment, try again, and so on.)

The remaining options specify features of the pdf document:

  • SHOW GRID has the effect that the table schema is underlaid by coordinates and gridlines. This is helpful during the layout phase.

  • SHOW COLOR relates to the color of the lines joining tables. (When colored lines are printed on a black-and-white printer, the results are not always lovely.)

  • DISPLAY TABLE DIMENSIONS results in every table's size being specified in the coordinate system of the pdf document. This option is useful only during the layout phase.

  • DISPLAY ALL TABLES WITH SAME WIDTH? relates to the width of tables in the pdf document. A uniform width is often easier to read.

  • DATA DICTIONARY specifies whether in addition to the table structure, a detailed description of the properties of all tables should be integrated into the pdf document (one page per table).

An OK allows the pdf document to be generated and transmitted to the web browser. Depending on the browser's settings, you can display it at once with Adobe Acrobat, or else you have to save it first to a local file. Figure 4-24 shows the settings for the table structure of mylibrary (Figure 4-25).

Click To expand
Figure 4-24: Setting the layout of the pdf table schema
Click To expand
Figure 4-25: The resulting structure as a pdf document


Chapter 5: Database Design

Overview

The first stage in any database application is the design of the database. The design will have a great influence on the efficiency of the application, the degree of simplicity or difficulty in programming and maintenance, and the flexibility in making changes in the design. Errors that occur in the design phase will come home to roost in the heartache of future efforts at correction.

This chapter discusses the fundamentals of relational databases, collects the varieties of data and tables available under MySQL, offers concrete examples of MySQL database structures, shows how indexes enable more efficient table access, and, finally, offers enlightenment on concrete ways that databases and tables can be generated.


Introduction

Database design is doubtless one of the greatest challenges in the development of complex database applications. This chapter attempts to provide a first introduction to the topic. But please don't expect any foolproof recipes. Many of the skills in database design are won by experience, and in a single chapter we can transmit only some of the basic principles.

To help in your orientation in this relatively long chapter, here is an overview:

  • The following two sections provide basic information. We discuss how to divide data logically among several tables, where the primary goal is to avoid redundancy. The normalization process helps in this.

    We also present the data types belonging to MySQL. For example, we answer the question as to what data type should be used for storing amounts of money and for long and short character strings.

  • We then go into some more advanced topics: How can searching and sorting tables be speeded up with the help of indexes? What types of tables does MySQL recognize, and which should be used when?

  • How are databases and their tables actually generated? A separate section introduces various CREATE commands, which help in this. (However, it is considerably more convenient to create new tables using instead a user interface such as phpMyAdmin.)

  • Theory is all well and good, but at some point we have to roll up our sleeves, put on an apron, and start cooking up some databases of our own. Therefore, the last sections of this chapter offer some concrete examples of the design of databases: for managing a small library, for running a discussion group, and for testing special cases.

    Pointer

    In this chapter we cannot avoid using some SQL commands now and then, although SQL is actually introduced in detail only in the next chapter. (However, it is conversely also impossible to describe SQL without assuming a particular database layout.) You may find it necessary at times to flip back and forth between this chapter and the next.

Further Reading

There are countless books that deal exclusively, independently of any specific database system, with database design and SQL. Needless to say, there is a variety of opinion as to which of these books are the good ones. Therefore, consider the following recommendations as my personal hit list:

  1. Joe Celko: SQL for Smarties, Morgan Kaufmann Publishers, 1999. (This is not a book for SQL beginners. Many examples are currently not realizable in MySQL, because MySQL is not sufficiently compatible with ANSI-SQL/92. Nonetheless, it is a terrific example-oriented book on SQL.)

  2. Judith S. Bowman et al.: The Practical SQL Handbook,Addison-Wesley, 2001.

  3. Michael J. Hernandez: Database Design for Mere Mortals, Addison-Wesley, 2003. (The first half is somewhat long-winded, but the second half is excellent and very clearly written.)

  4. The MySQL documentation recommends an additional book as an introduction to SQL, though I am not acquainted with it: Martin Gruber's Mastering SQL, Sybex, 2000.

  5. Another book that is frequently recommended is Peter Gulutzan and Trudy Pelzer's SQL-99 Complete, Really,R&D Books, 1999.

    Pointer

    If you are not quite ready to shell out your hard-earned (or perhaps even ill-gotten) cash for yet another book and you are interested for now in database design only, you may find the compact introduction on the design of relational databases by Fernando Lozano adequate to your needs. See http://www.edm2.com/0612/msql7.html.


Database Theory

Why is that that authors of books think about nothing (well, almost nothing) but books? The author begs the indulgence of his readers in that the example in this section deals with books. The goal of the section is to create a small database in which data about books can be stored: book title, publisher, author(s), publication date, and so on.

Normal Forms

These data can, of course, be stored without a database, in a simple list in text format, for example, as appears in the Bibliography at the end of this book:

Michael Kofler: Linux.Addison-Wesley 2000.

Michael Kofler, David Kramer: Definitive Guide to Excel VBA, second edition. Apress 2003.

Robert Orfali, Dan Harkey, Jeri Edwards: Client/Server Survival Guide. Addison-Wesley 1997.

Tobias Ratschiller, Till Gerken: Web Application Development with PHP 4.0. New Riders 2000.

This is a nice and convenient list, containing all necessary information. Why bother with all the effort to transform this text (which perhaps exists as a document composed with some word-processing program) into a database?

Needless to say, the reasons are legion. Our list can be easily searched, but it is impossible to organize it in a different way, for example, to create a list of all books by author x or to create a new list ordered not by author, but by title.

Our First Attempt

Just think: Nothing is easier than to turn our list into a database table. (To save space we are going to abbreviate the book titles and names of authors.)

Table 5-1 immediately shows itself to be riddled with problems. A first glance reveals that limiting the number of authors to three was an arbitrary decision. What do you do with a book that has four or five authors? Do we just keep adding columns, up to authorN, painfully aware that those columns will be empty for most entries?

Table 5-1: A book database: first attempt.
title | publisher | year | author1 | author2 | author3
Linux | Addison-Wesley | 2000 | Kofler, M. |  |
Definitive . . . | Apress | 2000 | Kofler, M. | Kramer, D. |
Client . . . | Addison-Wesley | 1997 | Orfali, R. | Harkey, D. | Edwards, E.
Web . . . | New Riders | 2000 | Ratschiller, T. | Gerken, T. |
The First Normal Form

Database theorists have found, I am happy to report, a solution to such problems. Simply apply to your database, one after the other, the rules for the three normal forms. The rules for the first normal form are as follows (though for the benefit of the reader, they have been translated from the language of database theorists into what we might frivolously call "linguistic normal form," or, more simply, plain English):

  • Columns with similar content must be eliminated.

  • A table must be created for each group of associated data.

  • Each data record must be identifiable by means of a primary key.

In our example, the first rule is clearly applicable to the authorN columns. The second rule seems not to be applicable here, since in our example we are dealing exclusively with data that pertain specifically to the books in the database. Thus a single table would seem to suffice. (We will see, however, that this is not, in fact, the case.)

The third rule signifies in practice that a running index must be used that uniquely identifies each row of the table. (It is not strictly necessary that an integer be used as primary key. Formally, only the uniqueness is required. For reasons of efficiency the primary key should be as small as possible, and thus an integer is generally more suitable than a character string of variable length.)

A reconfiguration of our table after application of the first and third rules might look like that depicted in Table 5-2.

Table 5-2: A book database: first normal form
id | title | publisher | year | author
1 | Linux | Addison-Wesley | 2000 | Kofler, M.
2 | Definitive Guide . . . | Apress | 2000 | Kofler, M.
3 | Definitive Guide . . . | Apress | 2000 | Kramer, D.
4 | Client/Server . . . | Addison-Wesley | 1997 | Orfali, R.
5 | Client/Server . . . | Addison-Wesley | 1997 | Harkey, D.
6 | Client/Server . . . | Addison-Wesley | 1997 | Edwards, E.
7 | Web Application . . . | New Riders | 2000 | Ratschiller, T.
8 | Web Application . . . | New Riders | 2000 | Gerken, T.

Clearly, the problem of multiple columns for multiple authors has been eliminated. Regardless of the number of authors, they can all be stored in our table. Of course, there is no free luncheon, and the price of a meal here is rather high: The contents of the columns title, publisher, and year are repeated for each author. There must be a better way!

Second Normal Form

Here are the rules for the second normal form:

  • Whenever the contents of columns repeat themselves, this means that the table must be divided into several subtables.

  • These tables must be linked by foreign keys.

If you are new to the lingo of the database world, then the term foreign key probably seems a bit, well, foreign. A better word in everyday English would probably be cross reference, since a foreign key refers to a line in a different (hence foreign) table. For programmers, the word pointer would perhaps be more to the point, while in Internet jargon the term link would be appropriate.

In Table 5-2, we see that data are repeated in practically every column. The culprit of this redundancy is clearly the author column. Our first attempt to give the authors their very own table can be seen in Tables 5-3 and 5-4.

Table 5-3: title table: second normal form
titleID | title | publisher | year
1 | Linux | Addison-Wesley | 2000
2 | Definitive Guide . . . | Apress | 2000
3 | Client/Server . . . | Addison-Wesley | 1997
4 | Web Application . . . | New Riders | 2000
Table 5-4: author table: second normal form
authorID | titleID | author
1 | 1 | Kofler, M.
2 | 2 | Kofler, M.
3 | 2 | Kramer, D.
4 | 3 | Orfali, R.
5 | 3 | Harkey, D.
6 | 3 | Edwards, E.
7 | 4 | Ratschiller, T.
8 | 4 | Gerken, T.

In the author table, the first column, with its running authorID values, provides the primary key. The second column takes over the task of the foreign key. It points, or refers, to rows of the title table. For example, row 7 of the author table indicates that Ratschiller, T. is an author of the book with ID titleID=4, that is, the book Web Application ... .

Second Normal Form, Second Attempt

Our result could hardly be called optimal. In the author table, the name Kofler, M. appears twice. As the number of books in this database increases, the amount of such redundancy will increase as well, whenever an author has worked on more than one book.

The only solution is to split the authors table again and live without the titleID column. The information as to which book belongs to which author must be specified in yet a third table. These three tables are shown in Tables 5-5 through 5-7.

Table 5-5: title table: second normal form
titleID | title | publisher | year
1 | Linux | Addison-Wesley | 2000
2 | Definitive Guide . . . | Apress | 2000
3 | Client/Server . . . | Addison-Wesley | 1997
4 | Web Application . . . | New Riders | 2000
Table 5-6: author table: second normal form
authorID | author
1 | Kofler, M.
2 | Kramer, D.
3 | Orfali, R.
4 | Harkey, D.
5 | Edwards, E.
6 | Ratschiller, T.
7 | Gerken, T.
Table 5-7: rel_title_author table: second normal form
titleID | authorID
1 | 1
2 | 1
2 | 2
3 | 3
3 | 4
3 | 5
4 | 6
4 | 7

This is certainly the most difficult and abstract step, probably because a table of the form rel_title_author has no real-world content. Such a table would be completely unsuited for unautomated management. But then, computers are another matter altogether, being totally without consciousness, regardless of what the nuttier cognitive scientists have to say.

Once a computer has been provided with a suitable program, such as MySQL, it has no trouble at all processing such data. Suppose you would like to obtain a list of all authors of the book Client/Server ... . MySQL would first look in the title table to find out what titleID number is associated with this book. Then it would search in the rel_title_author table for data records containing this number. The associated authorID numbers then lead to the names of the authors.

Remark

It may haveoccurred to you to ask why inthe rel_title_author table there is no ID column, say, rel_title_author_ID. Usually, such a column is omitted, since the combination of titleID and authorID is already an optimal primary key. (Relational database systems permit such primary keys, those made up of several columns.)
Third Normal Form

The third normal form has a single rule, and here it is:

  • Columns that are not directly related to the primary key must be eliminated (that is, transplanted into a table of their own).

In the example under consideration, the column publisher appears in the title table. The set of publishers and the set of book titles are independent of one another and therefore should be separated. Of course, it should be noted that each title must be related to the information as to the publisher of that title, but it is not necessary that the entire name of the publisher be given. A foreign key (that is, a reference, a pointer, a link) suffices. See Tables 5-8 and 5-9.

Table 5-8: title table: third normal form
titleID | title | publisherID | year
1 | Linux | 1 | 2000
2 | Definitive Guide . . . | 2 | 2000
3 | Client/Server . . . | 1 | 1997
4 | Web Application . . . | 3 | 2000
Table 5-9: publisher table: third normal form
publisherID | publisher
1 | Addison-Wesley
2 | Apress
3 | New Riders

The author and rel_title_author tables remain the same in the third normal form. The completed book database now consists of four tables, as indicated in Figure 5-1.

Click To expand
Figure 5-1: The structure of the database books

If we had paid closer attention to the rules for the first normal form (associated data belong together in a table), we could, of course, have saved some of our intermediate attempts. But that would have diminished the pedagogical value of our example. In fact, in practice, it often occurs that only when test data are inserted and redundancies are noticed does it become clear how the tables need to be subdivided.

The Bestowal of Names

In our example, we have attempted to leave the names of fields unaltered from the beginning (before normalization) right through to the end, so as not to cause more confusion than absolutely necessary. The result is not quite optimal, and it would be worth going back and looking at the complete table structure and reconsidering what names should be given to the various fields. The following points provide some tips on the naming of tables and their columns (fields).

  • MySQL distinguishes between uppercase and lowercase in the naming of tables, but not in the naming of columns (see also Chapter 18). Thus it is important to pay attention to this case sensitivity at least in names of tables. (In the example databases in this book, we shall use only lowercase letters for the names of tables.)

  • Of course, field and table names should be as clear as possible. Field names like author and publisher are not particularly good role models. First of all, they coincide with table names, which can cause confusion, and second, name or authorsName or companyName would have been more precise.

  • A uniform pattern for naming fields will save many errors caused by haste. Whether you prefer authors_name or authorsName, choose one format and stick with it.

  • Likewise, you should consider how to deal with singular and plural forms. In our example, I could have named the tables title, author, and publisher just as easily titles, authors, and publishers. There is no rule as to what is right or wrong here, but it is certainly confusing if half of your tables use the singular, and the other half, the plural.

  • Finally, it is worth thinking about whether field names should provide information about the table. In our example there are fields called titleID, publisherID, and the like. Fields containing a primary key could as well be called id. This would satisfy the uniqueness criterion, since in SELECT queries that encompass several tables, the table name has often to be provided in any case. (SQL allows the format tablename.fieldname.) This could lead to confusing instructions like WHERE publisher.publisherID = title.publisherID.

On the other hand, many tables also contain foreign keys, and there the specification of the table name is unavoidable. (In the title table, for example, you could not have three columns all labeled id.) That would lead to publisher.id = title.publisherID, which is also not optimal. (You can see that there is no easy solution that works optimally for all cases.)

Pointer

From books another example database was created called mylibrary.The design of that database is described later in this chapter. It is better than the books database in the sense of the bestowal of names. The database mylibrary serves as the basis for many examples in this book, while books exists only for the purpose of introducing a bit of database theory.

More Theory ...

The three normal forms for relational databases were first formulated by the researcher E. F. Codd. They continue to form the basis for a branch of research that is concerned with the formal description of mathematical sets in general and of relational databases in particular.

Depending on what you read, you may come across three additional normal forms, which, however, are of much less significance in practice. The normal forms and their rules are described much more precisely than we have done. However, such descriptions are so teeming with such exotica as entities, attributes, and their ilk that the connection with relational databases can easily be lost.

If you are interested in further details on this topic, then you are encouraged to look into a good book on the subject of database design (see also the suggestions at the beginning of this chapter).

Less Theory ...

I have attempted to present the first three normal forms in as simple and example-oriented a way as possible, but perhaps even that was too theoretical. Actually, the normal forms are not necessarily helpful to database beginners, since the correct interpretation of the rules is often difficult. Here are some rules that will perhaps make those first baby steps less shaky:

  • Give yourself sufficient time to develop the database. (If at a later date you have to alter the database design, when the database is already stuffed with real data and there is already client code in existence, changes will take a great deal of time and effort.)

  • Avoid giving columns names with numbers, such as name1, name2, or object1, object2. There is almost certainly a better solution involving an additional table.

  • Immediately supply your database with some test data. Attempt to include as many special cases as possible. If you encounter redundancies, that is, columns in which the same content appears several times, this is usually a hint that you should break your table into two (or more) new tables.

  • Tryto understand the ideas of relations (see the next section).

  • A good database design cannot be obtained unless you have had some experience with SQL (see also next chapter). Only when you know and understand the range of SQL queries can you judge the consequences of the various ways of organizing your data.

  • Orient yourself using an example database (from this book or from another book on databases).

    Tip

    A good example of normalizing a database can be found at the following Internet location:

    http://www.phpbuilder.com/columns/barry20000731.php3

Normal Forms: Pro and Con

Normal forms are a means to an end, nothing more and nothing less. Normal forms should be a help in the design of databases, but they cannot replace human reasoning. Furthermore, it is not always a good idea to follow the normal form thoughtlessly, that is, to eliminate every redundancy.

  • Con: The input of new data, say, in a form on a web page, becomes more and more complex as the number of tables among which the data are distributed increases. This is true as much for the end user (who is led from one page to another) as for the programmer.

    Furthermore, for efficiency in queries, it is sometimes advantageous to allow for a bit of redundancy. Bringing together data from several tables is usually slower than reading data from a single table. This is true especially for databases that do not change much but that are frequently given complex queries to respond to. (In a special area of databases, the data warehouses, redundancy is often deliberately planned for in order to obtain better response times. The purpose of data warehouses is the analysis of complex data according to various criteria. However, MySQL is not a reasonable choice of database system for such tasks anyhow, and therefore, we shall not go more deeply into the particular features of this special application area.)

  • Pro: Redundancy is generally a waste of storage space. You may hold the opinion that in the era of hundred-gigabyte hard drives this is not an issue, but a large database will inevitably become a slow database (at the latest, when the database size exceeds the amount of RAM).

    As a rule, databases in normal form offer more flexible query options. (Unfortunately, one usually notices this only when a new form of data query or grouping is required, which often occurs months after the database has become operational.)

Relations

If you want to transform a database into normal form, you have to link a number of tables. These links are called relations in database-speak. At bottom, there are three possible relations between two tables:

1:1

In a one-to-one relation between two tables, each data record of the first table corresponds to precisely one data record of the second table and vice versa. Such relations are rare, since in such a case the information in both tables could as easily be stored in a single table.

1: n

In a one-to-many relation, a single record in the first table can correspond to several records in the second table (for example, a vendor can be associated with many orders). The converse may be impossible: A single order cannot, say, be filled by many vendors. Occasionally, one hears of an n-to-1 relation, but this is merely a 1-to-n relation from the opposite point of view.

n : m

Here a data record in the first table can be linked to several records in the second table, and vice versa. (For example, several articles can be included in a single order, while the same article may be included in several different orders. Another example is books and their authors. Several authors may have written a single book, while one author may have written several books.)

1:1 Relations

A one-to-one relation typically comes into being when a table is divided into two tables that use the same primary key. This situation is most easily grasped with the aid of an example. A table containing a corporation's personnel records contains a great deal of information: name, department, date of birth, date of employment, and so on. This table could be split into two tables, called, say, personnel and personnel_extra, where personnel contains the frequently queried and generally accessible data, while personnel_extra contains additional, less used and more private, data.

There are two possible reasons for such a division. One is the security aspect: It is simple to protect the table personnel_extra from general access. (For MySQL, this argument is less important, since access privileges can in any case be set separately for each column of a table.)

The other reason is that of speed. If a table contains many columns, of which only a few are required by most queries, it is more efficient to keep the frequently used columns in a single table. (In the ideal situation, the first table would contain exclusively columns of a given size. Such tables are more efficient to manage than tables whose columns are of variable size. An overview of the types of tables supported by MySQL can be found later in this chapter.)

The significant disadvantage of such a separation of tables is the added overhead of ensuring that the tables remain synchronized.

1: n Relations

One-to-many relations come into play whenever a particular field of a data record in a detail table can refer to various columns of another table (the master table).

The linkage takes place via key fields. The columns of the master table are identified by a primary key. The detail table contains a foreign key field, whose contents refer to the master table. Here are a few examples:

  • The normalization example of the previous section (book database): Here there is a one-to-many relation between the title and publisher tables. The table publisher is the master table with the primary key publisher.publisherID. The title table is the detail table, with the foreign key title.publisherID (see Figure 5-1).

    Each publisher (1) can publish several books (n).

  • A business application containing tables with orders: A detail table contains data on all processed orders. In this table, a foreign key field refers to the master table, with its list of all customers.

    Each customer (1) can execute many orders (n).

  • Discussion groups containing tables with messages: A detail table contains data on every contribution to a discussion group in place on the web site (title, text, date, author, group, etc.). Two possible master tables are a group table with a list of all discussion groups, and an author table with a list of all members of the web site who are allowed to make contributions to a discussion.

    Each author (1) can contribute to arbitrarily many discussions (n). Each discussion group (1) can contain arbitrarily many contributions (n).

  • A database containing tables of music CDs: A detail table contains data on every CD in the collection (title, performer, number of disks, etc.). Two possible master tables are a table containing a list of performers occurring in the database, and a recording label table with a list of recording companies. Each performer (1) can appear on arbitrarily many CDs (n). Each label (1) can market arbitrarily many CDs (n).

Remark

Often during the creation of a database, one attempts to give the same name to fields of two tables that will later be linked by a relation. This contributes to clarity, but it is not required.

The table diagrams in this book (such as shown in Figure 5-1) were created with Microsoft Access. For all of the technical drawbacks that Access offers on account of the file/server architecture vis-à-vis MySQL, the user interface is easy to use. There is hardly a program with which relation diagrams can be so quickly drawn. However, Access has the peculiarity of labeling 1 : n diagrams with 1 : ∞. Here 1 denotes the master table (primary key), while infinitely many n stands for the detail table (foreign key). Often, an arrow is used instead of this nomenclature.

It is also possible for the primary and foreign keys to be located in the same table. Then a data record in such a table can refer to another record in the same table. This is useful if a hierarchy is to be represented. Here are a few examples:

  • A table of personnel, in which each employee record (except for that of the top banana) refers to a field containing that individual's immediate supervisor.

  • Discussion groups, with tables with messages, in which each message refers to a field containing the next-higher message in the hierarchy (that is, the one to which the current message is responding).

  • A music database, containing tables with different types of music. Each style field refers to a field with the name of the genre of which the current style is a subset (for example, bebop within the category jazz, or string quartet within the category of chamber music; see http://www.mp3.com).

    Remark

    Relations within a table indeed allow for a very simple way to store a hierarchy, but the evaluation of such data is more complicated. For example, the query command SELECT does not allow for recursion, which is often necessary for the analysis of hierarchical relations. Consider carefully the types of queries that will be applied to your database before you heedlessly implement complex hierarchies based on internal table references.

    The database mylibrary introduced later in this chapter provides a concrete example of the representation of hierarchies.

n : m Relations

For n : m relations, it is necessary to add an auxiliary table to the two original tables so that the n : m relation can be reduced to two 1: n relations. Here are some examples:

  • Normalization example (books database): Here we have an n : m relation between book titles and authors. The relation is established by means of the rel_title_author table.

  • Business application, tables with orders: To establish a relation between an order and the articles included in the order, the auxiliary table specifies how many of article x are included in order y.

  • College administration, list of exams: To keep track of which student has passed which exam and when and with what grade, it is necessary to have a table that stands between the table of students and the table of exams.

Primary and Foreign Keys

Relations depend intimately on primary and foreign keys. This section provides a comprehensive explanation of these two topics and their application. Alas, we cannot avoid entirely a bit of an excursus into SQL commands, which are not formally introduced until the end of this chapter and the chapter following.

Primary Key

The job of the primary key is to locate, as fast as possible, a particular data record in a table (for example, to locate the record with id=314159 from a table of a million records). This operation must be carried out whenever data from several tables are assembled—in short, very often indeed.

With most database systems, including MySQL, it is also permitted to have primary keys that are formed from several fields of a table. Whether it is a single field or several that serve as primary key, the following properties should be satisfied:

  • The primary key must be unique. It is not permitted that two records have the same content in their primary key field.

  • The primary key should be compact, and there are two reasons for this: First, for the primary key it is necessary to maintain an index (the primary index) to maximize the speed of search (e.g., for id=314159). The more compact the primary field key, the more efficient the management of this index. Therefore, an integer is more suitable than a character string of variable length for use as a primary key field.

    Second, the content of the primary key field is used as a foreign key in other tables, and there, as well, it is efficient to have the foreign key as compact as possible. (Relations between tables are established not least to avoid wasted space on account of redundancies. This makes sense only if the use of key fields doesn't take up even more space.)

With most database systems it has become standard practice to use a 32- or 64-bit integer as primary key field, generated automatically in sequence (1, 2, 3, ... ) by the database system. Thus neither the programmer nor the user need be concerned how a new and unique primary key value is to be found for each new record.

In MySQL such fields are declared as follows:

CREATE TABLE  publisher
  (publisherID  INT NOT NULL AUTO_INCREMENT,
  othercolumns  . . .  ,
  PRIMARY KEY  (publisherID))

If we translate from SQL into English, what we have is this: The field publisherID is not permitted to contain NULL. Its contents are generated by the database (unless another value is explicitly inserted there). The field functions as a primary key; that is, MySQL creates an auxiliary index file to enable rapid search. Simultaneously, it is thereby ensured that the publisherID value is unique when new records are input (even if a particular publisherID is specified in the INSERT command).

For tables in which one expects to make many new entries or changes, one should usually use BIGINT (64-bit integer) instead of INT (32 bits).

Remark

The name of the primary key field plays no role. In this book we usually use id or tablenameID. Often, you will see combinations with no or nr (for "number") as, for example, in customerNr.

Pointer

To generate sequential numbers for primary keys automatically, in MySQL the column is usually created with the attribute AUTO_INCREMENT. Additional information on AUTO_INCREMENT can be found in the next section of this chapter.

Foreign Key

The task of the foreign key field is to refer to a record in the detail table. However, this reference comes into being only when a database query is formulated, for example, in the following form:

SELECT title.title, publisher.publisher FROM title, publisher
WHERE title.publisherID = publisher.publisherID
ORDER BY title

With this, an alphabetical list of all book titles is generated, in which the second column gives the publisher of the book. The result would look something like this:

title | publisher
Client/Server . . . | Addison-Wesley
Definitive Guide . . . | Apress
Linux | Addison-Wesley
Web Application . . . | New Riders

Decisive here is the clause WHERE title.publisherID = publisher.publisherID. It is here that the link between the tables is created.

On the other hand, in the declaration of a table the foreign key plays no particular role. For MySQL, a foreign key field is just another ordinary table field. There are no particular key words that must be employed. In particular, no index is necessary (there is practically never a search to find the contents of the foreign key). Of course, you would not be permitted to supply the attribute AUTO_INCREMENT. After all, you want to specify yourself the record to which the field refers. You need to take care, though, that the foreign key field is of the same data type as the type of the primary key field. Otherwise, the evaluation of the WHERE condition can be very slow.

CREATE TABLE title
  (othercolumns ... ,
  publisherID INT NOT NULL)

Whether you specify the attribute NOT NULL depends on the context. In most cases, NOT NULL is to be recommended in order to avoid at the outset the occurrence of incomplete data. However, if you wish to allow, for example, that in the book database a book could be entered that had no publisher, then you should do without NOT NULL.

Referential Integrity

If you delete the author Kofler from the author table in the example of the normalized database (see Tables 5-6 through 5-9 and Figure 5-1), you will encounter problems in many SQL queries that access the books Linux and Definitive Guide. The authorID number 1 specified in the rel_title_author table no longer exists in the author table. In database language, one would put it like this: The referential integrity of the database has been damaged.

As a database developer, it is your responsibility to see that such events cannot happen. Therefore, before deleting a data record you must always check whether there exists a reference to the record in question in another table.

Since one cannot always rely on programmers (and databases often must be altered by hand), many databases have rules for maintaining referential integrity. Such rules test at every change in the database whether any cross references between tables are affected. Depending on the declaration of the foreign key, there are then two possible consequences: Either the operation will simply not be executed (error message), or all affected records in dependent tables are deleted as well. Which modus operandi is to be preferred depends on the data themselves.

In MySQL 4.0, referential integrity can be automatically ensured only with InnoDB tables. Independent control of referential integrity for the more commonly deployed MyISAM tables is planned for MySQL 5.0.

With InnoDB tables, you can declare foreign key fields like this:

CREATE TABLE title
    (column1, column2,    . . .  ,
      publisherID  INT,
      KEY  (publisherID),
      FOREIGN KEY  (publisherID)  REFERENCES  publisher (publisherID)
)

This means that title.publisherID is a foreign key that refers to the primary key publisher.publisherID. With options such as ON DELETE, RESTRICT, and ON DELETE CASCADE, one may further specify how the database system is to respond to potential damage to its referential integrity.

Moreover, this instruction will then be correctly executed even if the table driver does not support integrity rules. In this case, however, the FOREIGN KEY information is ignored, and currently, not even saved. (At least this latter should change in the future: Even if MySQL does not concern itself with maintaining referential integrity, the corresponding table declarations should be maintained. This would be particularly practical for programs for formulating SQL queries and for graphical user interfaces, which then would know how tables are linked with one another.)

Pointer

In Chapter 8, SQL commands for declaring referential integrity for InnoDB tables will be discussed in detail.

Other Indexes

Pointer

The primary index, which is inseparably bound to the primary key, is a special case of an index. Indexes always have the task of speeding up searching and sorting in tables by particular columns. Other indexes supported by MySQL are introduced later in the chapter.


MySQL Data Types

Now that we have dealt with the first issue in database design, dividing the data among several tables in the most sensible way, in this section we go a level deeper. Here you will find information about the data types that MySQL provides for every field (every column). Should a character string stored in a field be of fixed or variable length? Should currency amounts be stored as fixed-point or floating-point numbers? How are binary objects handled?

Pointer

This section describes the data types, but not the CREATE TABLE command, with which tables are actually created. Information on how tables can be created, that is, how the database, having been designed, can actually be implemented, can be found later in this chapter.

Default Values and NULL

Note that for each column, in addition to the data type, you can specify additional attributes. The following list names the three most important of these that are available for all data types:

  • NOT NULL: The column may not contain the value NULL. This means that in saving a new data record an explicit value must be given (unless there is a default value for the column).

  • NULL: The column is allowed to contain the value NULL. This is the default setting if when the table was created, neither NOT NULL nor NULL was specified.

  • DEFAULT n or DEFAULT 'abc': If no value is specified for a data record, MySQL automatically provides the value n or the character string 'abc'. (Here is a tip for advanced users: If you would like MySQL to insert the default value for an INSERT command, you must specify for the column either no value at all, or an empty character string. It is not permitted to supply the value NULL. In that case, MySQL stores NULL or returns an error message if NULL is not allowed.)

Even if you do not explicitly provide a default value, in many instances, MySQL will itself provide one: NULL if NULL is allowed, and otherwise, 0 for a numerical column, an empty character string for VARCHAR, the date 0000-00-00 in the case of dates, the year 0000 for YEAR,or the first element of an ENUM enumeration.

Remark

Unfortunately, MySQL does not allow a function as default value. Thus it is not possible to specify DEFAULT RAND( ) if you want a random number to be entered automatically in a column.

Integers: xxxINT

TINYINT(M) | 8-bit integer (1 byte, - 128 to +127)
SMALLINT(M) | 16-bit integer (2 bytes, - 32,769 to +32,767)
MEDIUMINT | 24-bit integer (3 bytes, - 8,388,608 to +8,388,607)
INT, INTEGER(M) | 32-bit integer (4 bytes, - 2,147,483,648 to +2,147,483,647)
BIGINT(M) | 64-bit integer (8 bytes, ±9.22 × 1018)

With the INT data type, both positive and negative numbers are generally allowed. With the attribute UNSIGNED, the range can be restricted to the positive integers. But note that then subtraction returns UNSIGNED integers, which can lead to deceptive and confusing results.

With TINYINT, numbers between -128 and +127 are allowed. With the attribute UNSIGNED, the range is 0 to 255. If one attempts to store a value above or below the given range, MySQL simply replaces the input with the largest or, respectively, smallest permissible value.

Optionally, in the definition of an integer field, the desired column width (number of digits) can be specified, such as, for example, INT(4). This parameter is called M (for maximum display size) in the literature. It assists MySQL as well as various user interfaces in presenting query results in a readable format.

Remark

Note that with the INT data types, the M restricts neither the allowable range of numbers nor the possible number of digits. In spite of setting INT(4), for example, you can still store numbers greater than 9999. However, in certain rare cases (such as in complex queries for the evaluation of which MySQL constructs a temporary table), the numerical values in the temporary tables can be truncated, with incorrect results as a consequence.

AUTO_INCREMENT Integers

With the optional attribute AUTO_INCREMENT you can achieve for integers that MySQL automatically inserts a number that is 1 larger than the currently largest value in the column when a new record is created for the field in question. AUTO_INCREMENT is generally used in the definition of fields that are to serve as the primary key for a table.

The following rules hold for AUTO_INCREMENT:

  • This attribute is permitted only when one of the attributes NOT NULL, PRIMARY KEY,or UNIQUE is used as well.

  • It is not permitted for a table to possess more than one AUTO_INCREMENT column.

  • The automatic generation of an ID value functions only when in inserting a new data record with INSERT, a specific value or NULL is not specified. However, it is possible to generate a new data record with a specific ID value, provided that the value in question is not already in use.

  • If you want to find out the AUTO_INCREMENT value that a newly inserted data record has received, after executing the INSERT command (but within the same connection or transaction), execute the command SELECT LAST_INSERT_ID( ).

  • If the AUTO_INCREMENT counter reaches its maximal value, based on the selected integer format, it will not be increased further. No more insert operations are possible. With tables that experience many insert and delete commands, it can happen that the 32-bit INT range will become used up, even though there are many fewer than two billion records in the table. In such a case, use a BIGINT column.

Floating-Point Numbers: FLOAT and DOUBLE

FLOAT(M, D) | floating-point number, 8-place precision (4 bytes)
DOUBLE(M, D) | floating-point number, 16-place precision (8 bytes)
REAL(M, D) | Synonym for DOUBLE

Since version 3.23 of MySQL, the types FLOAT and DOUBLE correspond to the IEEE numerical types for single and double precision that are available in many programming languages.

Optionally, the number of digits in FLOAT and DOUBLE values can be set with the two parameters M and D. In that case, M specifies the number of digits before the decimal point, while D gives the number of places after the decimal point.

The parameter M does no more than assist in the formatting of numbers; it does not limit the permissible range of numbers. On the other hand, D has the effect of rounding numbers when they are stored. For example, if you attempt to save the number 123456.789877 in a field with the attribute DOUBLE(6,3), the number stored will, in fact, be 123456.790.

Remark

MySQL expects floating-point numbers in international notation, that is, with a decimal point, and not a comma (which is used in a number of European countries). Results of queries are always returned in this notation, and very large or very small values are expressed in scientific notation (e.g., 1.2345678901279e+017).

If you have your heartset on formatting floating-point numbers differently, you will have either to employ the function FORMAT in your SQL queries (though this function is of use only in the thousands groupings) or to carry out your formatting in the client programming language (that is, in PHP, Perl, etc.).

Fixed-Point Numbers: DECIMAL(P, S)

DECIMAL(p, s) | fixed-point number, saved as a character string; arbitrary number of digits (one byte per digit + 2 bytes overhead)
NUMERIC, DEC | synonym for DECIMAL

The integer type DECIMAL is recommended when rounding errors caused by the internal representation of numbers as FLOAT or DOUBLE are unacceptable, perhaps with currency values. Since the numbers are stored as character strings, the storage requirement is much greater. At the same time, the possible range of values is smaller, since exponential notation is ruled out.

The two parameters P and S specify the total number of digits (precision) and, respectively, the number of digits after the decimal point (scale). The range in the case of DECIMAL(6,3) is from 9999.999 to −999.999. This bizarre range results from the apparent fact that six places are reserved for the number plus an additional place for the minus sign. When the number is positive, the place for the minus sign can be commandeered to store another digit. If P and S are not specified, then MySQL automatically uses (10, 0), with the result that positive integers with eleven digits and negative integers with ten digits can be stored.

Date and Time: DATE, TIME, and DATETIME

DATE | date in the form '2003-12-31', range 1000-01-01 to 9999-12-31 (3 bytes)
TIME | time in the form '23:59:59', range ± 838 : 59 : 59 (3 bytes)
DATETIME | combination of DATE and TIME in the form '2003-12-31 23:59:59' (8 bytes)
YEAR | year 1900–2155 (1 byte)

With the DATE and DATETIME data types, only a limited amount of type checking takes place. Values between 0 and 12 for months, and between 0 and 31 for days, are generally allowed. However, it is the responsibility of the client program to provide correct data. (For example, 0 is a permissible value for a month or day, in order to provide the possibility of storing incomplete or unknown data.) As for the question that may have occurred to you—Why is it that DATETIME requires eight bytes, while DATE and TIME each require only three bytes?—the answer is that I have no idea.

MySQL returns results of queries in the form 2003-12-31. However, with INSERT and UPDATE it manages to deal with other formats, provided that the order year/month/day is adhered to and the values are numeric. If the year is given as a two-digit number, then the following interpretation is made: 70–99 becomes 1970–1999, while 00–69 becomes 2000–2069.

If query results are to be specially formatted, there are several MySQL functions available for processing date and time values. The most flexible of these is DATE_FORMAT, whose application is demonstrated in the following example:

SELECT DATE_FORMAT(birthdate, '%Y %M %e') FROM students
2000 September 3
2000 October 25
...

Time of the Most Recent Change: TIMESTAMP(M)

TIMESTAMP

date and time in the form 20031231235959 for times between 1970 and 2038 (4 bytes)

Among the data types for date and time, TIMESTAMP plays a particular role. Fields of this type are automatically updated whenever the record is altered, thereby reflecting the time of the last change. Fields of type TIMESTAMP are therefore usually employed only for internal management, not for the storage of "real" data, though such is possible.

For the automatic TIMESTAMP updating to function properly, either no explicit value is assigned to the field, or else the value NULL.In both cases MySQL itself inserts the current time.

In the declaration of a TIMESTAMP column, the desired column width is specified. The TIMESTAMP values that then result from queries will be truncated. (In the case of M =8, for example, the result will be a date without the time). Internally, however, the values continue to be stored in their complete form.

Remark

Since version 4.1, MySQL automatically returns TIMESTAMP columns in the form YYYY-MM-DD HH:MM:DD (instead of the earlier YYYYMMDDHHMMDD). This can result in incompatibilities in data processing. Append a zero if you wish to use the old form: SELECT ts+0 FROM table.

Warning

Do not use a TIMESTAMP column if you wish to store the date and time yourself. For that, one has the data type DATETIME. With TIMESTAMP columns the MySQL server automatically changes the content of the column every time a change is made. You can prevent this behavior only by explicitly giving the desired date and time with each UPDATE or INSERT command. (The following example assumes that the TIMESTAMP column is called ts.)

UPDATE tablename SET col='new value', ts=ts;

Sooner or later, you will forget ts=ts and will then have the incorrect date/time in your table.
Remark

Many database operations with particular client libraries (for example, with Connector/ODBC) function only when each table of the database displays a TIMESTAMP column. The time of the last update is often needed in the internal administration of data.

Character Strings

CHAR(n) | character string with specified length, maximum 255 bytes
VARCHAR(n) | character string with variable length, maximum 255 bytes
TINYTEXT | character string with variable length, maximum 255 bytes
TEXT | character string with variable length, maximum 216 - 1 = 65,535 bytes
MEDIUMTEXT | character string with variable length, maximum 224 - 1 = 16,777,215 bytes
LONGTEXT | character string with variable length, maximum 232 - 1 = 4,294,967,295 bytes

With CHAR, the length of a character string is strictly specified. For example, CHAR(20) demands 20 bytes in each record, regardless of the length of the character string actually stored. (Blank characters at the beginning of a character string are eliminated before storage. Short character strings are extended with blanks. These blank characters are automatically deleted when the data are read out, with the result that it is impossible to store a character string that actually has blank characters at the end.)

In contrast, the length of a character string of type VARCHAR or one of the four TEXT types is variable. The storage requirement depends on the actual length of the character string.

Remark

MySQL 4.0 uses the character set latin1 by default for all databases, where there is the simple rule of one byte per character for storage.

Since MySQL 4.1, on the other hand, each column of a table can have a particular character set assigned to it. This includes character sets for which the storage requirement varies according to the character, for example, Unicode UTF8. This has the consequence that in a VARCHAR(50) column with the UTF8 character set, you can store fifty ASCII characters (one byte per character), but only twenty-five special German characters (two bytes per character). Many Asian characters require even more bytes. Thus it is no longer possible to specify exactly how many characters can be stored in a column. It depends on the character set used.

Although VARCHAR and TINYTEXT, both of which can accept a character string up to a length of 255 characters, at first glance seem equivalent, there are, in fact, several features that distinguish one from the other:

  • The maximum number of characters in VARCHAR columns must be specified (in the range 0 to 255) when the table is declared. Character strings that are too long will be unceremoniously, without warning, truncated when they are stored.

    In contrast, with xxxTEXT columns one cannot specify a maximal length. (The only limit is the maximal length of the particular text type.)

  • In VARCHAR columns, as with CHAR columns, blank characters are deleted from the beginning of a character string. (This behavior is ANSI compliant only for CHAR columns, not for VARCHAR columns. This behavior may change in future versions of MySQL.)

    With xxxTEXT columns, character strings are stored exactly as they are input.

Columns of type CHAR and VARCHAR can optionally be given the attribute BINARY. They then behave essentially like BLOB columns (see below). The attribute BINARY can be useful when you store text (and not binary objects): What you achieve is that in sorting, it is exclusively the binary code of the characters that is considered (and not a particular sorting table). Thus case distinction is made (which otherwise would not be the case). The internal management of binary character strings is simpler and therefore faster than is the case with garden-variety character strings.

Remark

As we have already mentioned, MySQL supports Unicode character strings, beginning with version 4.1. Of course, in version 4.0 as well you can store Unicode character strings in TEXT as well as BLOB fields. MySQL, however, is incapable of sorting such tables correctly, and there can be problems in searching and comparing as well.

Binary Data (xxxBLOB)

TINYBLOB | binary data, variable length, max 255 bytes
BLOB | binary data, variable length, max 216 - 1 bytes (64 kilobytes)
MEDIUMBLOB | binary data, variable length, max 224 - 1 bytes (16 megabytes)
LONGBLOB | binary data, variable length, max 232 - 1 bytes (4 gigabytes)

For the storage of binary data there are four BLOB data types at your service, all of which display almost the same properties as the TEXT data types. (Recall that "BLOB" is an acronym for "binary large object.") The only difference is that text data are usually compared and sorted in text mode (case-insensitive), while binary data are sorted and compared according to their binary codes.

Remark

There is considerable disagreement as to whether large binary objects should even be stored in a database. The alternative would be to store the data (images, for example) in external files and provide links to these files in the database.

The advantage to using BLOBs is the resulting integration into the database (more security, simpler backups). The drawback is the usually significant slowdown. It is particularly disadvantageous that large and small data elements—strings, integers, etc.—on the one hand and BLOBs and long texts on the other must be stored all mixed together in a table file. The result is a slowdown in access to all of the data records.

Note as well that BLOBs in general can be read only as a whole. That is, it is impossible to read, say, the last 100 kilobytes of an 800 kilobyte BLOB. The entire BLOB must be transmitted.

Enumerations: ENUM, SET

ENUM | selects one from at most 65,535 character strings (1 or 2 bytes)
SET | combines at most 64 character strings (1–8 bytes)

MySQL offers the two special enumeration types ENUM and SET. With ENUM, you can manage a list of up to 65,535 character strings, ordered by a running index. Then, in a field, one of these character strings can be selected.

In queries involving comparison of character strings there is no case distinction. In addition to the predefined character strings, an empty character string can also be stored in a field (as well as NULL, unless this has been excluded via NOT NULL).

Such a field is then handled like any other character string field. The following commands show how a table with an ENUM enumeration is generated and used. In the field color of the table testenum, one of five predefined colors can be stored:

CREATE TABLE testenum
  (color ENUM ('red', 'green', 'blue', 'black', 'white'))
INSERT testenum VALUES ('red')
SELECT * FROM testenum WHERE color='red'

SET uses a similar idea, though here arbitrary combinations are possible. Internally, the character strings are ordered by powers of 2 (1, 2, 4, 8, etc.), so that a bitwise combination is possible. The storage requirement is correspondingly larger (one bit per character string). At most 64 character strings can be combined (in which case the storage requirement is 8 bytes).

Fora combination of several character strings to be stored in one field, these must be given separated by commas (and with no blank characters between strings). The order of the strings is irrelevant and is not considered. In query results, combinations are always specified in the order in which the set was defined:

CREATE TABLE testset
  (fontattr SET ('bold', 'italic', 'underlined'))
INSERT testset VALUES ('bold,italic')

In queries with the operator "=" an exact comparison is made of the entire combination. The result is that only those records are returned for which the combination corresponds exactly. Thus if in testset only the above-inserted record is stored with 'bold,italic', then the following query returns no result:

SELECT * FROM testset WHERE fontattr='italic'

In order to locate records in which an attribute has been set (regardless of its combination with other attributes), the MySQL function FIND_IN_SET can be used. This function returns the position of the sought character string within the set (in our example, 1 if 'bold' is found, 2 for 'italic', etc.):

SELECT * FROM testset WHERE FIND_IN_SET('italic', fontattr)>0
Tip

ENUM and SET values are represented internally as integers, not as character strings. If you wish to determine the internally stored value via a query, simply use SELECT x+0 FROM table, where x is the column name of the ENUM or SET column. It is also permitted to store numeric values with INSERT and UPDATE commands.
Remark

The contents of ENUM and SET fields are not alphabetically sorted, but are maintained in the order in which the character strings for selection were defined. The reason for this is that MySQL works internally with numeric values associated with the character strings. If you would like an alphabetic sorting, you must transform the string explicitly into a character string, for example, via SELECT CONCAT(x) AS xstr ... ORDER BY xstr.

Tip

If you would like to determine the list of all admissible character strings for an ENUM or SET field (in a client program, for example), you must summon DESCRIBE tablename columnname to your aid. This SQL command returns a table in which the field columnname is described. The column Type of this table contains the ENUM or SET definition. In Chapter 11 you will find an example for the evaluation of this information (in the programming language PHP).


Indexes

If you are searching for a particular record in a table or would like to create a series of data records for an ordered table, MySQL must load all the records of the table. The following lines show some of the relevant SELECT commands (details to follow in the next chapter):

SELECT  column1, column2 . . . FROM  table  WHERE  column3=12345
SELECT  column1, column2 . . . FROM  table  ORDER BY  column3
SELECT  column1, column2 . . . FROM  table  WHERE  column3  LIKE  'Smith%'
SELECT  column1, column2 . . . FROM  table  WHERE  column3 > 2000

With large tables, performance will suffer under such everyday queries. Fortunately, there is a simple solution to cure our table's performance anxiety: Simply use an index for the affected column (in the example above, for column3).

An index is a special file or, in the case of InnoDB, a part of the tablespace, containing references to all the records of a table. (Thus a database index functions like the index in this book. The index saves you the trouble of reading the entire book from one end to the other if you simply want to find out where a particular topic is covered.)

In principle, an index can be created for each field of a table, up to a maximum of sixteen indexes per table. (MySQL also permits indexes for several fields simultaneously. That makes sense if sorting is frequently carried out according to a combination of fields, as in WHERE country='Austria' AND city='Graz').

Caution

Indexes are not a panacea! They speed up access to data, but they slow down each alteration in the database. Every time a data record in changed, the index must be updated. This drawback can be ameliorated to some extent with various SQL commands by means of the option DELAY_KEY_WRITE.The effect of this option is that the index is not updated with each new or changed record, but only now and then. DELAY_KEY_WRITE is useful, for example, when many new records are to be inserted in a table as quickly as possible.

A further apparent disadvantage of indexes is that they take up additional space on the hard drive. (Internally, in addition, B trees are used for managing the index entries.)

Therefore, use indexes only for those columns that will often be searched and sorted. Indexes remain largely useless when the column contains many identical entries. (In such cases, you might ask yourself whether the normalization of the database has been optimally carried out.)

Types of Index

All indexes are not alike, and in this section we discuss the various types of indexes.

Ordinary Index

The only task of an ordinary index (defined via the key word INDEX) is to speed up access to data.

Tip

Index only those columns that you require in conditions (WHERE column= ... ) or forsorting (ORDER BY column). Index, if possible, columns with compact data (for example, integers). Do not index columns that contain many identical values. (For example, it makes little sense to index a column with 0/1 or Y/N values.)

Restrictions

MySQL cannot use indexes where inequality operators are used (WHERE column != ... ). Likewise, indexes cannot be used for comparisons where the contents of the column are processed by a function (WHERE DAY(column)= ... ).

With JOIN operations (that is, in uniting data from various tables), indexes are of use only when primary and foreign keys refer to the same data type.

If the comparison operators LIKE and REGEXP are used, an index is of use only when there is no wild card at the beginning of the search pattern. With LIKE 'abc%' an index is of use, but with LIKE '%abc',itis not.

Finally, indexes are used with ORDER BY operations only if the records do not have to be previously selected by other criteria. (Unfortunately, an index rarely helps to speed up ORDER BY with queries in which the records are taken from several tables.)

Unique Index

With an ordinary index it is allowed for several data records in the indexed field to refer to the same value. (In a table of personnel, for example, the same name can appear twice, even though it refers to two distinct individuals.)

When it is clear from context that a column contains unique values, you should then define an index with the key word UNIQUE. This has two consequences. One is that MySQL has an easier time managing the index; that is, the index is more efficient. The other is that MySQL ensures that no new record is added if there is already another record that refers to the same value in the indexed field. (Often, a UNIQUE index is defined for this reason alone, that is, not for access optimization, but to avoid duplication.)

Primary Index

The primary index mentioned again and again in this chapter is nothing more than an ordinary UNIQUE index. The only peculiarity is that the index has the name PRIMARY.

(You can tell that a primary index is an ordinary index because MySQL uses the first UNIQUE index of a table as a substitute for the missing primary index.)

Combined Indexes

An index can cover several columns, as in INDEX(columnA, columnB). A peculiarity of such indexes is that MySQL can selectively use such an index. Thus when a query requires an index for columnA only, the combined index for INDEX(columnA, columnB) can be used. This holds, however, only for partial indexes at the beginning of the series. For instance, INDEX(A, B, C) can be used as index for A or (A, B), but not as index for B or C or (B,C).

Limits on the Index Length

In the definition of an index for CHAR andVARCHAR columns you can limit an index to a particular number of characters (which must be smaller than the maximum number of characters allowed in this field). The consequence is that the resulting index file is smaller and its evaluation quicker than otherwise. In most applications, that is, with character strings representing names, perhaps ten to fifteen characters altogether suffice to reduce the search set to a few data records.

With BLOB and TEXT columns you must institute this restriction, where MySQL permits a maximal index length of 255 characters.

Full-Text Index

An ordinary index for text fields helps only in the search for character strings that stand at the beginning of the field (that is, whose initial letters are known). On the other hand, if you store texts in fields that consist of several, or possibly very many, words, an ordinary index is useless. The search must be formulated in the form LIKE '%word%', which for MySQL is rather complex and with large data sets leads to long response times.

In such cases it helps to use a full-text index. With this type of index, MySQL creates a list of all words that appear in the text. A full-text index can be created during the database design or afterwards:

ALTER TABLE tablename ADD FULLTEXT(column1, column2)

In SELECT queries, one can now search for records that contain one or more words. This is the query syntax:

SELECT * FROM tablename
WHERE MATCH(column1, column2) AGAINST('word1', 'word2', 'word3')

Then all records will be found for which the words word1, word2, and word3 appear in the columns column1 and column2.

Pointer

An extensive description of the SQL syntax for full-text search, together with a host of application examples, can be found in Chapter 7.

Query and Index Optimization

Realistic performance estimates can be made only when the database has been filled with a sufficient quantity of test data. A test database with several hundred data records will usually be located entirely in RAM after the first three queries, and all queries will be answered quickly with or without an index. Things become interesting when tables contain well over 1000 records and when the entire size of the database is larger than the total RAM of the MySQL server.

In making the decision as to which columns should be provided with indexes, one may sometimes obtain some assistance from the command EXPLAIN SELECT. This is simply an ordinary SELECT command prefixed with the key word EXPLAIN.Instead of the SELECT being simply executed, MySQL places information in a table as to how the query was executed and which indexes (to the extent that they exist) came into play.

Here are some pointers for interpreting the table created by EXPLAIN. In the first column appear the names of the tables in the order in which they were read from the database. The column type specifies how the table is linked to the other tables (JOIN). This functions most efficiently (i.e., quickly) with the type system, while more costly are the types const, eq_ref, ref, range, index, and ALL. (ALL means that for each record in the next-higher table in the hierarchy, all records of this table must be read. That can usually be prevented with an index. Further information on all JOIN types can by found in the MySQL documentation.)

The column possible_keys specifies which indexes MySQL can access in the search for data records. The column key specifies which index MySQL has actually chosen. The length of the index in bytes is given by key_len. For example, with an index for an INTEGER column, the number of bytes is 4. Information on how many parts of a multipart index are used is also given by key_len. As a rule, the smaller key_len is, the better (that is, the faster).

The column ref specifies the column of a second table with which the linkage was made, while rows contains an estimate of how many records MySQL expects to read in order to execute the entire query. The product of all the numbers in the rows column allows one to draw a conclusion as to how many combinations arise from the query.

Finally, the column extra provides additional information on the JOIN operation, for example, using temporary when MySQL must create a temporary table in executing a query.

Pointer

Though the information proffered by EXPLAIN is often useful, the interpretation requires a certain amount of MySQL and database experience. You will find further information in the MySQL documentation:

http://www.mysql.com/doc/en/Query_Speed.html
http://www.mysql.com/doc/en/EXPLAIN.html

Example 1

This query produces an unordered list of all books with all their authors. All ID columns are equipped with primary indexes.

USE mylibrary
EXPLAIN SELECT * FROM titles, rel_title_author, authors
  WHERE rel_title_author.authID = authors.authID
  AND rel_title_author.titleID = titles.titleID
table | type | possible_key | key_len | ref | rows
titles | ALL | PRIMARY |  |  | 23
rel_title_author | ref | PRIMARY PRIMARY | 4 | titles.titleID | 1
authors | eq_ref | PRIMARY PRIMARY | 4 | rel_title_author.authID | 1

This means that first all records from the titles table are read, without using an index. For titles.titleID there is, in fact, an index, but since the query considers all records in any case, the index is not used. Then with the help of the primary indexes of rel_title_author and authors, the links to the two other tables are made. The tables are thus optimally indexed, and for each part of the query there are indexes available.

Example 2

Here the query produces a list of all books (together with their authors) that have been published by a particular publisher. The list is ordered by book title. Again, all ID columns are equipped with indexes. Furthermore, in the titles table, title and publID are indexed:

EXPLAIN SELECT title, authName
FROM titles, rel_title_author, authors
WHERE titles.publID=1
  AND titles.titleID = rel_title_author.titleID
  AND authors.authID = rel_title_author.authID
ORDER BY title
table | type | key | key_len | ref | rows | Extra
titles | ref | publIDIndex | 5 | const | 2 | Using where; Using filesort
rel_title_author | ref | PRIMARY | 4 | titles.titleID | 1 | Using index
authors | eq_ref | PRIMARY | 4 | rel_title_author.authID | 1 |

To save space in this example, we have not shown the column possible_keys. The interpretation is this: The tables are optimally indexed; that is, for each part of the query there are indexes available. It is interesting that the title list (ORDER BY title) is apparently sorted externally, although there is an index for the title column as well. The reason for this is perhaps that the title records are first selected in accordance with the condition publID=2, and the title index can then no longer be applied.

Example 3

This example uses the same SELECT query as does Example 2, but it assumes that titles.publID does not have an index. The result is that now all 36 records of the titles table must be read, and no index can be used:

table | type | key | key_len | ref | rows | Extra
titles | ALL |  |  |  | 36 | Using where; Using filesort
rel_title_author | ref | PRIMARY | 4 | titles.titleID | 1 | Using index
authors | eq_ref | PRIMARY | 4 | rel_title_author.authID | 1 |


MySQL Table Types

Up till now, we have tacitly assumed that all tables have been set up as MyISAM tables. That has been the default behavior since MySQL 3.23, and it holds automatically if you do not explicitly demand a different type of table when the table is generated. This section provides a brief overview of the different types of table recognized by MySQL, their properties, and when they should be used. (The MySQL server can, however, be configured using the option default-table-type to make another table type valid.)

  • MyISAM—Static: These tables are used when all columns of the table have fixed, predetermined size. Access in such tables is particularly efficient. This is true even if the table is frequently changed (that is, when there are many INSERT, UPDATE, and DELETE commands). Moreover, data security is quite high, since in the case of corrupted files or other problems, it is relatively easy to extract records.

  • MyISAM—Dynamic: If in the declaration of a table there is also only a single VARCHAR, xxxTEXT, or xxxBLOB field, then MySQL automatically selects this table type. The significant advantage over the static MyISAM variant is that the space requirement is usually significantly less: Character strings and binary objects require space commensurate with their actual size (plus a few bytes overhead).

    However, it is a fact that data records are not all the same size. If records are later altered, then their location within the database file may have to change. In the old place there appears a hole in the database file. Moreover, it is possible that the fields of a record are not all stored within a contiguous block within the database file, but in various locations. All of this results in increasingly longer access times as the edited table becomes more and more fragmented, unless an OPTIMIZE TABLE or an optimization program is executed every now and then (myisamchk; see Chapter 10).

  • MyISAM—Compressed: Both dynamic and static MyISAM tables can be compressed with the auxiliary program myisamchk. This usually results in a shrinkage of the storage requirement for the table to less than one-half the original amount (depending on the contents of the table). To be sure, thereafter, every data record must be decompressed when it is read, but it is still possible under some conditions that access to the table is nevertheless faster, particularly with a combination of a slow hard drive and fast processor.

    The decisive drawback of compressed MyISAM tables is that they cannot be changed (that is, they are read-only tables).

  • ISAM tables: The ISAM table type is the precursor of MyISAM, and is supported only for reasons of compatibility. ISAM tables exhibit a host of disadvantages with respect to MyISAM tables. Perhaps the most significant of these is that the resulting files are not independent of the operating system. Thus in general, an ISAM table that was generated by MySQL under operating system x cannot be used under operating system y. (Instead, you must use mysqldump and mysql to transfer it to the new operating system.) In general, there is no longer any good reason to use ISAM tables (unless, perhaps, you are using an antiquated version of MySQL).

  • InnoDB tables: The InnoDB table type is the most important alternative to the MyISAM tables. The use of such tables involves a number of different advantages and disadvantages.

    • Advantages vis-à-vis MyISAM: InnoDB tables support transactions, which ensures the integrity of multistep database operations. InnoDB tables also support functions for ensuring referential integrity. ◦

    • Disadvantages vis-à-vis MyISAM: All InnoDB tables are stored in a single binary file, which is thus to some extent a black box. To move a single table or database from one server to another requires the use of mysqldump. (With MyISAM tables you can simply copy and transfer the file.) Moreover, InnoDB tables do not yet support a full-text index. The storage requirements on the hard disk are greater than those of comparable MyISAM tables.

    Elementary database operations are generally slower with InnoDB tables than with MyISAM tables. InnoDB tables can, however, greatly improve their performance speed if through the use of transactions one can do without LOCK commands.

    In general, many InnoDB functions are similar to those in Oracle tables. The InnoDB table type has been developed over a number of years and is used in a large number of database systems. You will find extensive information on InnoDB in Chapter 8.

  • BDB (Berkeley_DB): BDB tables also support transactions. (It was the BDB table type that first made transactions possible under MySQL.) In comparison to InnoDB, the integration into MySQL is not as good, and the documentation in a number of details is rather vague. For these reasons, the BDB table driver is not integrated into the standard version of the MySQL server. As a rule, it is advisable to use InnoDB tables in preference to BDB tables.

  • Gemini: In 2001, the Nusphere company made available its own version of MySQL, distinguished by the addition of the Gemini table type, which supports transactions. Unfortunately, a legal battle developed between Nusphere and the MySQL company, a battle over whether Nusphere had adhered to the rules of the MySQL license (GPL). The disagreement was settled in November 2002, with the result that the code for the Gemini table type was not added to the official MySQL code.

  • MERGE: MERGE tables are essentially a virtual union of several existing MyISAM tables all of which exhibit identical column definitions. A MERGE table composed of several tables can have some advantages over a single, large, MyISAM table, such as a higher read speed (if the tables are distributed over several hard drives) or a circumvention of the maximum file size in a number of older operating systems (for example, 2 gigabytes for Linux; 2.2 for 32-bit processors). Among the disadvantages are that it is impossible to insert data records into MERGE tables (that is, INSERT does not function). Instead, INSERT must be applied to one of the subtables.

    In the meanwhile, since most modern operating systems support files of arbitrary size, as well as RAID (that is, the division of a file system on several hard disks), MERGE tables play a subordinate role in practice.

  • HEAP: HEAP tables exist only in RAM (not on the hard drive). They use a hash index, which results in particularly fast access to individual data records.

    In comparison to normal tables, HEAP tables present a large number of functional restrictions, of which we mention here only the most important: No xxxTEXT or xxxBLOB data types can be used. Records can be searched only with = or <=> (and not with <, >, <=, >=). AUTO_INCREMENT is not supported. Indexes can be set up only for NOT NULL columns.

    HEAP tables should be used whenever relatively small data sets are to be managed with maximal speed. Since HEAP tables are stored exclusively in RAM, they disappear as soon as MySQL is terminated. The maximum size of a HEAP table is determined by the parameter max_heap_table_size, which can be set when MySQL is launched.

    Remark

    From among the table types listed above, only the most important will be discussed in this book: MyISAM, InnoDB, and HEAP. Further information on the other table types can be found in the MySQL documentation:

    http://www.mysql.com/doc/en/Table_types.html

Temporary Tables

With most of the table types listed above there exists the possibility of creating a table on a temporary basis. Such tables are automatically deleted as soon as the link with MySQL is terminated. Furthermore, temporary tables are invisible to other MySQL links (so that it is possible for two users to employ temporary tables with the same name without running into trouble).

Temporary tables are not a separate table type unto themselves, but rather a variant of the types that we have been describing. Temporary tables are often created automatically by MySQL in order to assist in the execution of SELECT queries.

Temporary tables are not stored in the same directory as the other MySQL tables, but in a special temporary directory (under Windows it is usually called C:\Windows\Temp, while under Unix it is generally /tmp or /var/tmp or /usr/tmp). The directory can be set at MySQL launch.

Remark

Often, temporary tables are generated that are of type HEAP. However, this is not a necessary combination. Note, however, that nontemporary HEAP tables are visible to all MySQL users until they are deleted.

Table Files

The location of database files can be set when MySQL is launched. (Under Unix/Linux, /var/lib/mysql is frequently used, while under Windows it is usually C:\mysql\data.) All further input is given relative to this directory.

Where MySQL tables are actually stored depends very much on the table type. Basically, the description of each table is stored in a *.frm file, and these *.frm files are organized in directories that correspond to the names of the databases:

data/dbname/tablename.frm

for the table structure (data type of the columns, indexes, etc.).

For each MyISAM table, two additional files are created:

data/dbname/table name.MYD

for the MyISAM table data, and

data/dbname/tablename.MYI

for the MyISAM table indexes (all the indexes of the table).

InnoDB tables, on the other hand, are stored together in a single file or in several associated files. These files form the so-called tablespace, a sort of virtual storage for all tables. The name and location of these files can be prescribed by configuration settings. By default,

data/ibdata1, 2, 3

hold the InnoDB tables (data and indexes), while

data/ib_logfile0, 1, 2

hold the InnodDB logging files.

Starting with MySQL 4.1, an additional file, db.opt, is stored in the database directory. It contains settings that hold for the entire database:

data/dbname/db.opt


Creating Databases, Tables, and Indexes

In the course of this chapter you have learned a great deal about database design. But building castles in the air is one thing, and assembling the plans and materials for real-world construction quite another. We now confront the question of how a database and its tables are actually built. As always, there are several ways of going about this task:

  • Genuine SQL pros will not shrink before the prospect of doing things the old-fashioned way and typing in a host of CREATE TABLE commands in mysql. However, the complicated syntax of this command is not compatible with the creative character of the design process. Nonetheless, SQL commands are often the only way to achieve certain features of tables.

  • For beginners in the wonderful world of database construction (and for all who do not enjoy making their lives more complicated than necessary), the graphical user interfaces offer more convenience in the development of a database. Some of these programs (for example, the MySQL Control Center and phpMyAdmin) have been introduced in the previous chapter.

In this chapter, only the CREATE TABLE variant is considered. All of our examples refer to the little book database books, which in the portion of this chapter dealing with theory was introduced for the purpose of demonstrating the normalization process.

Remark

Regardless of the tool that you choose to work with, before you begin, the question of access rights must be clarified. If MySQL is securely configured and you do not happen to be the MySQL administrator, then you are not permitted to create anewdatabase at all.

The topics of access privileges and security are not dealt with until Chapter 9. If you are responsible for the security of MySQL, then you should probably concern yourself with this issue before you create your new database. If there is an administrator, then please ask him or her to set up an empty database for you and provide you with a user name and password.
Pointer

This section assumes that the new database will be created on the local computer, which is plausible if one is creating a new application. What happens, then, when a web application will run on the Internet service provider's computer? How do you transport the database from the local computer to that of the ISP? How can you, should the occasion arise, create a new database directly on the ISP's computer? Such questions will be discussed in their own section of Chapter 10.

Pointer

There is yet another path to a MySQL database: the conversion of an existing database from another system. Fortunately, there are quite a few converters available, and they can be found at the MySQL web site, which at last sighting was at the following addresses:

http://www.mysql.com/doc/en/Contrib.html
http://www.mysql.com/portal/software/index.html

In this book only two variants will be considered, namely, conversion of databases in the format of Microsoft Access or of Microsoft SQL Server. This topic is dealt with in Chapter 17.

Creating a Database (CREATE DATABASE)

Before you can create tables for your database, you must first create the database itself. The creation of a database results in an empty directory being created on the hard drive. The following USE command makes the new database the default database for all further SQL commands:

CREATE DATABASE books
USE books
Remark

The commands given here can be executed, for example, in the MySQL monitor mysql. (Please recall that in mysql you must follow each command with a semicolon.)

A complete syntax reference for the SQL commands introduced here can be found in Chapter 18.

Creating Tables (CREATE TABLE)

For the four tables of our example database, we require four CREATE TABLE commands. Each of these commands contains a list of all fields (columns) with the associated data type and attributes. The syntax should be immediately clear. The definition of the primary index is effected with PRIMARY KEY, where all columns that are to be included in the index are specified in parentheses. Of interest here is the table rel_title_author,in which the primary index spans both columns:

CREATE TABLE author (
   authorID      INT             NOT NULL     AUTO_INCREMENT,
   author        VARCHAR(60)     NOT NULL,
   PRIMARY KEY   (authorID))
CREATE TABLE publisher (
   publisherID   INT             NOT NULL     AUTO_INCREMENT,
   publisher     VARCHAR(60)     NOT NULL,
   PRIMARY KEY   (publisherID))
CREATE TABLE rel_title_author (
   titleID       INT             NOT NULL,
   authorID      INT             NOT NULL,
   PRIMARY KEY   (titleID, authorID))
CREATE TABLE title (
   titleID       INT             NOT NULL AUTO_INCREMENT,
   title         VARCHAR(120)    NOT NULL,
   publisherID   INT             NOT NULL,
   year          INT             NOT NULL,
   PRIMARY KEY   (titleID))

Editing a Table Design (ALTER TABLE)

It is fortunate indeed that you do not have to recreate your database from scratch every time you wish to modify its design, say to add a new column or change the data type or default value of a column. The command ALTER TABLE offers sufficient flexibility in most cases. It is also good to know that MySQL is capable in most cases of preserving existing data (even if the data type of a column is changed). Nevertheless, it is a good idea to maintain a backup copy of the table in question.

The following example shows how the maximal number of characters in the title column of the title table is reduced to 100 characters. (Titles longer than 100 characters will be truncated.) The first line of code shows how to make a backup of the original table (which is a good idea, since we might lose some data if titles are truncated):

CREATE TABLE titlebackup SELECT * FROM title
ALTER TABLE title CHANGE title title VARCHAR(100)
NOT NULL

Something that might be a bit confusing with our use of the ALTER TABLE command is the threefold appearance of the name title. The first usage refers to the table, while the second is the name of the column before the change, and the third the new, in this case unchanged, name of the column.

Inserting Data (INSERT)

Normally, data are inserted into the database via a client program (which, it is to be hoped, is convenient and easy to use). Since you, as developer, are responsible in most cases for the code of these programs, it would not hurt for us to take a peek at the INSERT command. (Additional commands for inserting, changing, and deleting data can be found in the next chapter.)

INSERT INTO author VALUES ( '1', 'Kofler M.')
INSERT INTO author VALUES ('2', 'Kramer D.')
...
INSERT INTO publisher VALUES ('1', 'Addison-Wesley')
INSERT INTO publisher VALUES ('2', 'Apress')
...
INSERT INTO title VALUES ('1', 'Linux', '1', '2000')
INSERT INTO title VALUES ('2', 'Definitive Guide to Excel VBA', '2', '2000')
...
INSERT INTO rel_title_author VALUES ('1', '1')
INSERT INTO rel_title_author VALUES ('2', '1')
INSERT INTO rel_title_author VALUES ('2', '2')
...

You can also edit several records at once:

INSERT INTO rel_title_author VALUES ('1', '1'), ('2', '1'), ('2', '2'), ...

Creating Indexes (CREATE INDEX)

Indexes can be created via the CREATE TABLE command or else later with an ALTER TABLE or CREATE INDEX command. The syntax for describing the index is the same in all three cases. The following three commands show three variants for providing the title column of the title table with an index. The index received the name idxtitle. (Of course, only one of the three commands can be executed. If you try a second command, MySQL will inform you that an index already exists.)

CREATE TABLE title (
  titleID . . . , title . . . , publisherID . . . , year . . . , PRIMARY KEY . . . ,
  INDEX idxtitle (title))
CREATE INDEX idxtitle ON title (title)
ALTER TABLE title ADD INDEX idxtitle (title)

SHOW INDEX FROM tablename produces a list of all defined indexes. Existing indexes can be eliminated with DROP INDEX indexname ON tablename.

If you wish to reduce the number of significant characters per index in the index to the first 16 characters, the syntax looks like this:

ALTER TABLE title ADD INDEX idxtitle (title(16))


Example mylibrary (Library)

Writers of books frequently not only write books, but read them as well (as do many other individuals who do not write books), and over time, many of these readers acquire a significant collection of books, sometimes so many that they lose track of what's what and what's where. After a hopeless effort at bringing order to the author's domestic library, he created the database mylibrary. SeeFigure 5-2 for an overview of the structure of the mylibrary database. The database mylibrary is an extended version of the database books that was introduced at the beginning of this chapter. Note that the names of individual tables and columns have been changed in the migration from books to library.

The main purpose of this database is to store the titles of books together with their authors and publishers in a convenient format. The database will additionally offer the convenient feature of allowing books to be ordered in various categories.

Click To expand
Figure 5-2: Structure of the mylibrary database
Tip

If you happen to be a victim of the compulsion to collect something, you can use mylibrary as a basis for your own database. You should have little trouble in converting this database into one for CDs, MP3 files, stamps, scientific articles, postal and e-mail addresses, Barbie dolls, birthdays and other noteworthy dates, arachnids, and so on.
Remark

The database mylibrary as well as its variants mylibraryodbc and mylibraryinno,areavailable at www.apress.com.

For mylibararyodbc, all tables are equipped with an additional timestamp column. This column is necessary if the database is to be altered with ODBC programs (see Chapter 17).

For mylibraryinno, alltables have been transformed into the InnoDB format. Furthermore, integrity rules (foreign key constraints have been defined.

Basic Design

Titles, Authors, and Publishers

Just as in the books database, here there are three tables in which book titles, names of authors, and names of publishers are stored: titles, authors, and publishers.

The n : m relation between titles and authors is created via the table rel_title_author. Here the sequence of authors for a given book can be specified in the field authNr (that is, the first, second, third, etc., author). This is practical above all if the database is to store the names of scientific publications, where it is not always a good idea to list the authors in alphabetical order. However, authNr can be left empty (that is, NULL)ifyou are not interested in such supplemental information.

In general, in the library database it is permitted to store the value NULL in many of the columns. Thus it is possible to store a book title quickly without bothering to specify the publisher or year of publication. Perfectionists in database design would, of course, frown on such practice, but on the other hand, too many restrictions often lead to the result that data simply do not get catalogued. From this point of view, it is practical to allow the publisher, say, to be indicated for some books without it being mandatory that it be given for all.

Categories and Languages

The table categories enables the construction of a hierarchical category list in which the books can be ordered. (Details on this table appear in the following section.) The table languages contains a list with the languages in which the books have appeared. Thus for each title, the language can be stored as well.

Dealing with (Author) Names

In mylibrary, names are generally stored in a field where the surname must be given first (to take a random example, Kofler Michael, and not Michael Kofler). This is the usual ordering for sorting by name.

A quite different approach would be to save surnames, given names, and even initials and titles, in separate fields (for example, firstname, lastname, and perhaps even middlename, etc.). Several fields are, of course, more trouble to manage than a single field, but in return, a greater flexibility is achieved as to how the name can be displayed. This is important, for example, if the data are to be used for personalizing letters or e-mail messages.

Independent of the internal storage, there is clearly a problem in that many people (myself included, alas) now and again input the surname and given name in the wrong order. One falsely input name will be practically untraceable in the database.

Therefore, it is important that the user interface always show and receive names in a uniform order and that the desired order—best indicated by an example—be shown in the input form. Be sure to label the input fields in a clear and unambiguous way (for example, use "family name" or "surname" and not "last name").

Data Types

All tables are provided with an AUTO_INCREMENT column, which also serves as the primary index. Most fields possess the attribute NOT NULL.Exceptions are those fields that describe the properties of the title. Thus NULL is allowed for subtitle, edition, publID, catID, langID, year, isbn, and comment.

Column name | Data type
All ID fields | INT
authors.authName | VARCHAR(60)
categories.catName | VARCHAR(60)
categories.hierNr | INT
categories.hierIndent | TINYINT
languages.langName | VARCHAR(40)
publishers.publName | VARCHAR(60)
titles.title and .subtitle | VARCHAR(100)
titles.year | INT
titles.edition | TINYINT
titles.isbn | VARCHAR(20)
titles.comment | VARCHAR(255)

Hierarchical Categorization of the Books

The table categories helps in ordering books into various categories (technical books, children's books, etc.). To make the library more interesting from the point of view of database design, the field parentCatID will be represented hierarchically. Here is an example: Figure 5-3 shows how the hierarchy represented in Table 5-10 can be depicted with respect to the database.

Figure 5-3: Example data for the categories table
Table 5-10: Database representation of the hierarchy in Figure 5-3
catID | catName | parentCatID
1 | Computer books | 11
2 | Databases | 1
3 | Programming | 1
4 | Relational databases | 2
5 | Object-oriented databases | 2
6 | PHP | 3
7 | Perl | 3
8 | SQL | 2
9 | Children's books | 11
10 | Literature and fiction | 11
11 | All books | NULL
Remark

In the development of applications, parentCatID=NULL must be treated as a special case.

Hierarchy Problems

Let me state at once that although the representation of such hierarchies looks simple and elegant at first glance, they cause many problems as well. (That this example nonetheless uses hierarchies has, of course, pedagogical purposes: It allows us to demonstrate interesting SQL programming techniques. Even though the representation of hierarchies in this example is somewhat artificial, there are many database applications where the use of hierarchies is unavoidable.)

Pointer

In point of fact, this chapter is supposed to be dealing with database design and not with SQL, but these two topics cannot be cleanly separated. There is no point in creating a super database design if the capabilities of SQL do not suffice to extract the desired data from the database's tables.

If you havenoexperiencewithSQL,youshould dip into the following chapter a bit. Consider the following instructions as a sort of "advanced database design."

Almost all problems with hierarchies have to do with the fact that SQL does not permit recursive queries:

  • With individual queries it is impossible to find all categories lying above a given category in the hierarchy.

    Example: The root category is called Relational databases (parentCatID=2). You would like to create a list that contains Computer books ® Databases ®Relational databases.

    With SELECT * FROM categories WHERE catID=2, you indeed find Databases, but not Computer books, which lies two places up the hierarchy. For that, you must execute an additional query SELECT * FROM categories WHERE catID=1. Of course, this can be accomplished in a loop in the programming language of your choice (Perl, PHP, etc.), but not with a single SQL instruction.

  • It is just as difficult to represent the entire table in hierarchical form (as a tree). Again, you must execute a number of queries.

  • It is not possible without extra effort to search for all books in a higher category.

    Example: You would like to find all books in the category Computer books.

    With SELECT * FROM titles WHERE catID=1 you find only those titles directly linked to the category Computer books, but not the titles in the categories Databases, Relational databases, Object-oriented databases, etc. The query must be the following: SELECT * FROM titles WHERE catID IN (1, 2 ... ), where 1, 2 ... are the ID numbers of the subordinate categories. The actual problem is to determine these numbers.

  • In the relatively simple representation that we have chosen, it is not possible to associate the same subcategory with two or more higher-ranking categories.

    Example: The programming language SQL is linked in the above hierarchy to the higher-ranking category Databases. It would be just as logical to have a link to Programming. Therefore, it would be optimal to have SQL appear as a subcategory of both Databases and Programming.

  • There is the danger of circular references. Such references can, of course, appear only as a result of input error, but where there are human beings who input data (or who write programs), there are certain to be errors. If a circular reference is created, most database programs will find themselves in an infinite loop. The resolution of such problems can be difficult.

None of these problems is insuperable. However, hierarchies often lead to situations in which answering a relatively simple question involves executing a whole series of SQL queries, and that is often a slow process. Many problems can be avoided by doing without genuine hierarchies (for example, by allowing at most a two-stage hierarchy) or if supplementary information for a simpler resolution of hierarchies is provided in additional columns or tables (see the following section).

Optimization (Efficiency Versus Normalization)

If you have taken to heart the subject of normalization of databases as presented in this chapter, then you know that redundancy is bad. It leads to unnecessary usage of storage space, management issues when changes are made, etc.

Yet there are cases in which redundancy is quite consciously sought in order to increase the efficiency of an application. The mylibrary database contains many examples of this, which we shall discuss here rather fully.

This section should make clear that database design is a multifaceted subject. There are usually several ways that lead to the same goal, and each of these paths is, in fact, a compromise of one sort or another. Which compromise is best depends largely on the uses to which the database will be put: What types of queries will occur most frequently? Will data be frequently changed?

Authors' Names in the titles Table

The authors' names are stored in the authors table in normalized form. However, it will often happen that you wish to obtain a list of book titles, and with each book title you would like to list all of that book's authors, such as in the following form:

Client/Server Survival Guide by Robert Orfali, Dan Harkey, Jeri Edwards

Definitive Guide to Excel VBA by Michael Kofler, David Kramer

Linux by Michael Kofler

Web Application Development with PHP 4.0 by Tobias Ratschiller, Till Gerken

There are now two possible ways of proceeding:

  • Variant 1: The simplest and thus most often chosen route is first to execute an SQL query to determine the desired list of titles, without the corresponding authors. Then for each title found, a further SQL query is formulated to determine all the authors of that title. Now, however, twenty-one SQL queries were necessary to obtain a list with twenty book titles. The following lines show the requisite pseudocode. The bits in pointy brackets cannot be executed in

    SQL; they must be formulated in a client programming language (Perl, PHP, etc.).

    SELECT titleID, title FROM titles WHERE . . . _for each titleID_
  SELECT authName FROM authors, rel_title_author
  WHERE rel_title_author.authID = authors.authID
  AND rel_title_author.titleID = _titleID_
  ORDER BY authNR

  • Variant 2: Another possibility is to include all the data in a single query:

    SELECT titles.titleID, title, authName
FROM titles, rel_title_author, authors
WHERE rel_title_author.titleID = titles.titleID
AND rel_title_author.authID = authors.authID
ORDER BY title, rel_title_author.authNr, authName

    You thereby obtain, however, a list in which each book title appears as many times as it has authors (see Table 5-11).

    Table 5-11: Query result in the mylibrary database

    titleID

    title

    authName

    3

    Client/Server Survival Guide

    Edwards E.

    3

    Client/Server Survival Guide

    Harkey D.

    3

    Client/Server Survival Guide

    Orfali R.

    2

    Definitive Guide to Excel VBA

    Kofler Michael

    2

    Definitive Guide to Excel VBA

    Kramer David

    1

    Linux, 5th ed.

    Kofler Michael

    9

    MySQL & mSQL

    King T.

    9

    MySQL & mSQL

    Reese G.

    9

    MySQL & mSQL

    Yarger R.J.

    4

    Web Application Development with PHP 4.0

    Gerken Till

    4

    Web Application Development with PHP 4.0

    Ratschiller Tobias

    At this point, you can use some client-side code to create the list of books by collecting authors' names for books with the same titleID number and displaying the book title only once.

    The significant disadvantage of this modus operandi is that it is relatively difficult to represent the results in page format (for example, with twenty titles per page). Of course, you can limit the number of result records in the SELECT query with LIMIT, but you don't know in advance how many records will be required to obtain twenty titles (since it depends on the number of authors per title).

    This problem can be circumvented if first a query is executed by which only the titleID values are determined. The second query then uses these titleID values. Unfortunately, MySQL 4.0 does not yet permit nested SELECT queries, and therefore either the titleID list must be processed by the client program in the second query (WHERE titleID IN(x, y, ... ), or a temporary table must be used. Beginning with MySQL 4.1, such queries can be formulated as subSELECTs.

  • Solution: From the standpoint of efficiency, the second variant is significantly better than the first, but even it is not wholly satisfying. The most efficient solution would be to provide a column authorsList in the titles table, in which the list of authors is stored for each title.

    Such a column is no substitute for the authors table, which will be used and reused (for example, to search for all books on which a particular author has worked, or to store additional information about an author). The column authorsList must be updated each time a new title is stored or an old one is changed. It is much more burdensome that now title records must also be updated when changes in the authors table are made (such as when a typo in an author's name is corrected): a classical redundancy problem.

    You thus have the choice between redundancy and efficiency. Which variant is better for your application depends on the size of the database, whether it will be used primarily in read-only mode for queries or frequently altered, what type of queries will be used most frequently, etc. (In our realization of mylibrary, we have done without authorsList.)

Hierarchical Order in the categories Table

The necessity will continually arise to display the categories table in hierarchical representation similar to that of Figure 5-3. As we have already mentioned, such processing of the data is connected either with countless SQL queries or complex client-side code. Both of these are unacceptable if the hierarchical representation is needed frequently (for example, for a listbox in a form).

A possible solution is provided by the two additional columns hierNr and hierIndent. The first of these gives the row number in which the record would be located in a hierarchical representation. (The assumption is that data records are sorted alphabetically by catName within a level of the hierarchy.) The second of these two columns determines the level of indentation. In Table 5-12 are displayed for both of these columns the values corresponding to the representation in Figure 5-3.

Table 5-12: categories table with hierNr column
catID | catName | parentCatID | hierNr | hierIndent
1 | Computer books | 11 | 2 | 1
2 | Databases | 1 | 3 | 2
3 | Programming | 1 | 7 | 2
4 | Relational databases | 2 | 5 | 3
5 | Object-oriented databases | 2 | 4 | 3
6 | PHP | 3 | 9 | 3
7 | Perl | 3 | 8 | 3
8 | SQL | 2 | 6 | 3
9 | Children's books | 11 | 1 | 1
10 | Literature and fiction | 11 | 10 | 1
11 | All books | NULL | 0 | 0

A simple query in mysql proves that this arrangement makes sense. Here are a few remarks on the SQL functions used: CONCAT joins two character strings. SPACE generates the specified number of blank characters. AS gives the entire expression the new Alias name category:

SELECT CONCAT(SPACE(hierIndent*2), catName) AS category
FROM categories ORDER BY  hierNr
category
All books
  Children's
  Computer books
   Databases
     Object-oriented databases
     Relational databases
     SQL
   Programming
     Perl
     PHP
  Literature and fiction

You may now ask how the numerical values hierIndent and hierNr actually come into existence. The following example-oriented instructions show how a new data record (the computer book category Operating systems)is inserted into the table:

  1. The data of the higher-ranking initial record (that is, Computer books) are known: catID=1, parentCatID=11, hierNr=1, hierIndent=1.

    1. Now we search within the Computer books group for the first record that lies in the hierarchy immediately after the record to be newly inserted (here, this is Programming). See Figure 5-3. All that is of interest in this record is hierNr.

      Here is a brief explanation of the SQL command:

      WHERE parentCat_ID=1 finds all records that are immediately below Computer books in the hierarchy (that is, Databases and Programming).

      catName>'Operating Systems' restricts the list to those records that occur after the new record Operating Systems.

      ORDER BY catname sorts the records that are found.

      LIMIT 1 reduces the result to the first record.

      SELECT hierNr FROM categories
WHERE parentCatID=1 AND catName>'Operating Systems'
ORDER BY catName
LIMIT 1

      The query just given returns the result hierNr=7. It is thereby clear that the new data record should receive this hierarchy number. First, however, all existing records with hierNr>=7 should have their values of hierNr increased by 1.

    2. It can also happen that the query returns no result, namely, when there are no entries in the higher-ranking category or when all entries come before the new one in alphabetic order. (This would be the case if you wished to insert the new computer book category Software engineering.)

      In that case, you must search for the next record whose hierNr is larger than hierNr for the initial record and whose hierIndent is less than or equal to hierIndent of the initial record. (In this way, the beginning of the next equal- or higher-ranking group in the hierarchy is sought.)

      SELECT hierNr FROM categories
WHERE hierNr>1 AND hierIndent<=1
ORDER BY hierNr LIMIT 1

      This query returns the result 10 (that is, hierNr for the record Literature and fiction). The new record will get this hierarchy number. All existing records with hierNr>=10 must have their hierNr increased by 1.

    3. If this query also returns no result, then the new record must be inserted at the end of the hierarchy list. The current largest hierNr value can easily be determined:

      SELECT MAX(hiernr) FROM categories

  3. To increase the hierNr of the existing records, the following command is executed (for case 2a):

    UPDATE categories SET hierNr=hierNr+1 WHERE hiernr>=7

  4. Now the new record can be inserted. For parentCatID, the initial record catID will be used. Above, hierNr=7 was determined. Here hierIndent must be larger by1 than was the case with the initial record:

    INSERT INTO categories (catName, parentCatID, hierNr, hierIndent)
VALUES ('Operating systems', 1, 7, 2)

The description of this algorithm proves that inserting a data record is a relatively complex and costly process. (A concrete realization of the algorithm can be found in the form of a PHP script in Chapter 12.) It is much more complicated to alter the hierarchy after the fact. Imagine that you wish to change the name of one of the categories in such a way as to change its place in the alphabetical order. This would affect not only the record itself, but many other records as well. For large sections of the table it will be necessary to determine hierNr. You see, therefore, that redundancy is bad.

Nevertheless, the advantages probably outweigh the drawbacks, above all because changes in the category list will be executed very infrequently (and thus the expenditure of time becomes insignificant), while queries to determine the hierarchical order, on the other hand, must be executed frequently.

Searching for Lower-Ranked Categories in the categories Table

Suppose you want to search for all Databases titles in the titles table. Then it would not suffice to search for all titles with catId=2, since you also want to see all the titles relating to Relational databases, Object-oriented databases, and SQL (that is, the titles with catID equal to 4, 5, and 8). The totality of all these categories will be described in the following search category group.

There are two problems to be solved: First, you must determine the list of the catID values for the search category group. For this a series of SELECT queries is needed, and we shall not go into that further here. Then you must determine from the titles table those records whose catID numbers agree with the values just found. Since MySQL, alas, does not support subSELECTs, for the second step a temporary table will be necessary.

Thus in principle, the title search can be carried out, but the path is thorny, with the necessity of several SQL queries and client-side code.

The other solution consists in introducing a new (redundant) table, in which are stored all records lying above each of the categories records. This table could be called rel_cat_parent, and it would consist of two columns: catID and parentID (see Table 5-13). We see, then, for example, that the category Relational databases (catID=4) lies under the categories All books, Computer books, and Databases (parentID=11, 1, 2).

Table 5-13: Some entries in the rel_cat_parent table
catID | parentID
1 | 11
2 | 1
2 | 11
3 | 1
3 | 11
4 | 1
4 | 2
4 | 11
. . . | . . .

The significant drawback of the rel_cat_parent table is that it must be synchronized with every change in the categories table. But that is relatively easy to take care of.

In exchange for that effort, now the question of all categories ranked below Databases is easily answered:

SELECT catID FROM rel_cat_parent
WHERE parentID=2

If youwould like to determine all book titles that belong to the category Databases or its subcategories, the requisite query looks like the following. The key word DISTINCT is necessary here, since otherwise, the query would return many titles with multiplicity:

SELECT DISTINCT titles.title FROM titles, rel_cat_parent
WHERE (rel_cat_parent.parentID = 2 OR titles.catID = 2)
  AND titles.catID = rel_cat_parent.catID

We are once more caught on the horns of the efficiency versus normalization dilemma: The entire table rel_cat_parent contains nothing but data that can be determined directly from categories. In the concrete realization of the mylibrary database, we decided to do without rel_cat_parent, because the associated PHP example (see Chapter 12) in any case does not plan for category searches.

Searching for Higher-Ranked Categories in the categories Table

Here we confront the converse question to that posed in the last section: What are the higher-ranking categories above an initial, given, category? If the initial category is Perl (catID=7), then the higher-ranking categories are first Programming, then Computer books, and finally, All books.

When there is a table rel_cat_parent like that described above, then our question can be answered by a simple query:

SELECT CONCAT(SPACE(hierIndent*2), catName)AS category
FROM categories, rel_cat_parent
WHERE rel_cat_parent.catID = 7
  AND categories.catID = rel_cat_parent.parentID
ORDER BY hierNr

The result is seen in Figure 5-4.

Figure 5-4: Query result in search for higher-ranking categories

On the other hand, if rel_cat_parent is not available, then a series of SELECT instructions must be executed in a loop categories.parentCatID until this contains the value 0. This process is demonstrated in the PHP example program categories.php (see Chapter 12).

Indexes

All the mylibrary tables are supplied with a PRIMARY index for the ID column. Additionally, the following usual indexes are defined:

Table | Column | Purpose
authors | author | search/sort by author's name
categories | catName | search by category name
 | hierNr | hierarchical sort of the category list
publishers | publName | search/sort by publisher's name
titles | title | search/sort by book title
 | publID | search by titles from a particular publisher
 | catID | search by titles from a particular category
 | isbn | search by ISBN


Example myforum (Discussion Group)

Among the best-loved MySQL applications are guest books, discussion groups, and other web sites that offer users the possibility of creating a text and thereby adding their own voices to the web site. The database myforum creates the basis for a discussion group. The database consists of three tables:

  • forums contains a list of the names of all discussion groups. Furthermore, each group can be assigned a particular language (English, German, Zemblan, etc.).

  • users contains a list of all users registered in the database who are permitted to contribute to discussions. For each registered user, a login name, password, and e-mail address are stored. (One can imagine extending this to hold additional information.)

  • messages contains all the stored contributions. These consist of a Subject row, the actual text, forumID, userID, and additional management information. This table is the most interesting of the three from the standpoint of database design.

The structure of the myforum database is depicted in Figure 5-5.

Click To expand
Figure 5-5: Structure of the myforum database
Remark

Included in the code files for this book are two versions of the myforum database. The file myforum.sql contains the SQL instructions for creating a test version of myforum that contains only test contributions.

In contrast, bigforum.sql contains a larger version (18 forums, 1400 contributions) that is the result of the discussion group on my web site. For this book, contributions have been made anonymous. (The users table contains only first names.) The database bigforum offers the possibility of trying out various MySQL operations (such as full-text search) on a database that is more than just a toy example.

The bigforum database contains messages in both English and German. However, the majority are in German. If you want to see only the English entries, you can build your queries thus:

SELECT * FROM messages, forums
WHERE messages.forumID=forums.forumID AND forums.language="english"

Data Types

All three tables are equipped with AUTO_INCREMENT columns, which also serve as primary index. In general, all fields have the attribute NOT NULL. Here we list the column names and data types of the three tables:

Column name | Data type
All ID fields | INT
forums.forumname | VARCHAR(80)
forums.language | ENUM('german', 'english', 'zemblan')
messages.subject | VARCHAR(80)
messages.msgText | TEXT
messages.level | INT
messages.timest | TIMESTAMP
messages.orderstr | VARCHAR(128), BINARY
users.username | VARCHAR(30)
users.email | VARCHAR(120)
users.password | VARCHAR(30)

Hierarchies Among Messages

As we have already seen in the case of the mylibrary database, there is a battle in myforum in dealing with hierarchies. A significant feature of discussion groups is that the discussion thread is represented in a hierarchical list. The hierarchy is the result of the fact that each contribution can elicit a response. A discussion among the five participants Antony, Banquo, Coriolanus, Duncan, and Edmund (A, B, C, D, and E for short) might look like that depicted in Figure 5-6.

Click To expand
Figure 5-6: A discussion thread

Here as illustration a description of the content is added to the title line (subject) of each contribution. In reality, the title line of a response is usually simply the letters Re: together with the original title line (e.g., Re: How do I ... ).

A representation in database format might look like that shown in Table 5-14. It is assumed here that the five participants A through E have userID numbers 201 through 205. The messages of the thread have msgID numbers that begin with 301. (In practice, it is natural to expect that a thread will not exhibit sequential msgID numbers. More likely, other threads will break into the sequence.) The table is sorted chronologically (that is, in the order in which the messages were posted).

Table 5-14: messages table with records of a discussion thread
msgID | userID | subject | parentID | timest
301 | 201 | How do I . . . | 0 | 2001-01-17 12:00
302 | 202 | first answer | 301 | 2001-01-17 18:30
303 | 201 | thanks | 302 | 2001-01-17 19:45
304 | 204 | second answer | 301 | 2001-01-18 3:45
305 | 201 | I don't . . . | 304 | 2001-01-18 9:45
306 | 205 | third answer | 301 | 2001-01-18 19:00
307 | 204 | the same . . . | 305 | 2001-01-18 22:05
308 | 203 | better suggestion | 302 | 2001-01-19 10:30

The hierarchy is expressed via the column parentID, which refers to the message up one level in the hierarchy. If parentID has the value 0, then the contribution is one that began a new thread.

Optimization

From the standpoint of normalization, parentID is sufficient to express the hierarchy. However, this slick table design makes some frequently occurring queries difficult to answer:

  • What are all the messages in this discussion thread?

  • How should the messages be sorted? (A glance at Table 5-14 shows that a chronological sorting is not our goal.)

  • How far should each message be indented if the messages are to be represented hierarchically?

To answer the first question, it is necessary to search (in a recursive loop) through all parentID/msgID pairs. With long threads, this will require a large number of SELECT queries. Of course, each of these queries can be quickly executed, but taken together, they represent a considerable degree of inefficiency.

Recursion also comes into play in the second question: A sorting by the time stored in timest leads to our goal, but the answers must be sorted by group for each branch of the hierarchical tree and inserted at the correct place in the list.

The answer to the third question is achieved by following the parentID pointer back to the start of the discussion thread.

To increase efficiency, we must back off from our wishful goal of zero redundancy. We shall add three new columns:

  • rootID refers to the original message that begins the discussion thread and thus makes it possible to answer the first question easily.

  • orderstr contains a binary concatenation of the timest times of all higher-ranking messages. As we shall soon see, this character string can be used for sorting messages within a thread.

  • level specifies the level of the hierarchy and thus immediately answers the third question.

Having absorbed these new columns into our database, we have a new representation of our database, as depicted in Table 5-15. In comparison with Table 5-14, you can see in the column timest that now there are only three digits, which should correspond to the internal binary representation. (In fact, four bytes are used, which permits 256 different values. For greater clarity we have used three-digit numbers here.)

Table 5-15: messages table with records of a discussion thread
msgID | userID | subject | parentID | timest | rootID | level | orderstr
301 | 201 | How . . . | 0 | 712 | 0 | 0 | 712
302 | 202 | first . . . | 301 | 718 | 301 | 1 | 712718
303 | 201 | thanks | 302 | 719 | 301 | 2 | 712718719
304 | 204 | second . . . | 301 | 803 | 301 | 1 | 712803
305 | 201 | I don't . . . | 304 | 809 | 301 | 2 | 712803809
306 | 205 | third . . . | 301 | 819 | 301 | 1 | 712819
307 | 204 | the same. . . | 305 | 822 | 301 | 3 | 712803809822
308 | 203 | better . . . | 302 | 910 | 301 | 2 | 712718910

While the meaning of the contents of the columns rootID and level is clear, that of orderstr might be a bit more opaque. In this binary column is stored a binary sequence of TIMESTAMP values of all higher-ranking messages. This simple measure ensures that associated groups of messages are not separated during sorting.

Pluses and Minuses

The obvious advantage of the three additional columns is the increased efficiency in displaying the messages. Since the reading operation occurs much more frequently than the writing operation in a discussion forum, reading efficiency has a correspondingly high status in the design of our database. (Furthermore, saving a new data record is efficient and not at all problematic. The values for the three supplementary columns can be easily determined for a response from the associated higher-ranking message. If the message begins a new thread, then rootID and level are 0, while orderstr contains the binary representation of the current time.)

The single drawback is the additional storage space required for the three columns. However, in comparison to the typical storage requirement of a message with several lines of text, this additional storage requirement can be considered negligible. (The maximum length of orderstr is limited to 128 characters. That is, at most 32 hierarchical levels can be represented.)

Due to the nature of discussion groups, the additional disadvantages due to redundancy, that is, the problems of subsequent changes in data, are insignificant. Changes in existing messages are not anticipated, and neither are additions and deletions after the fact, which would have the effect of changing the sequence of messages.

Remark

Please note that the orderstr character string is based on TIMESTAMP values, which change only once per second. If it should chance that two messages are received at the same second, the sort order could become confused.

Indexes

All myforum tables are supplied with a PRIMARY index for the ID column. Furthermore, the following usual indexes are defined:

Table | Column | Purpose
users | userName | search for user name (at login)
messages | orderStr | sort messages in discussion order
 | forumID | search for messages in a discussion group
 | rootID | search for messages in a thread

The PHP example for myforum does not provide for searching for messages. For such a search, a full-text index covering the columns subject and msgText is recommended.


Example exceptions (Special Cases)

When you begin to develop an application with a new database, programming language, or API that is unfamiliar to you it is often practical to implement a simple test database for quickly testing various special cases. For the work on this book, as well as for testing various APIs and various import and export tests, the database exceptions was used. Among the tables of this database are the following:

  • Columns with most data types supported by MySQL, including xxxTEXT, xxxBLOB, SET, and ENUM

  • NULL values

  • Texts and BLOBs with all possible special characters

  • All 255 text characters (code 1 to 255)

The following paragraphs provide an overview of the tables and their contents. The column names indicate the data type (thus the column a_blob has data type BLOB). The id column is an AUTO_INCREMENT column (type INT). In all the columns except the id column the value NULL is allowed.

testall

This table contains columns with the most important MySQL data types (though not all types).

Columns: id, a_char, a_text, a_blob, a_date, a_time, a_timestamp, a_float, a_decimal, a_enum, a_set

text_text

With this table you can test the use of text.

Columns: id, a_varchar (maximum 100 characters), a_text, a_tinytext, a_longtext

test_blob

With the test_blob table you can test the use of binary data (import, export, reading and storing a client program, etc.).

Columns: id, a_blob.

Contents: A record (id=1) with a 512-byte binary block. The binary data represent byte for byte the codes 0, 1, 2, ... , 255, 0, 1, ... , 255.

test_date

With this table you can test the use of dates and times.

Columns: id, a_date, a_time, a_datetime, a_timestamp.

Content: A data record (id=1) with the values 2000-12-07, 09:06:29, 2000-12-07 09:06:29, and 20001207090649.

test_enum

With this table you can test the use of SET and ENUM.

Columns: id, a_enum, and a_set (with the character strings 'a', 'b', 'c', 'd', 'e').

Content:

id | a_enum | a_set
1 | a | a
2 | e | b,c,d
3 |  |
4 | NULL | NULL

test_null

With this table you can test whether NULL (first record) can be distinguished from an empty character string (second record).

Columns: id, a_text.

Content:

id | a_text
1 | NULL
2 |
3 | 'a text'

test_order_by

With this table you can test ORDER BY and the character set in use (including sort order). The table consists of two columns: id and a_char. The column id contains sequential numbers from 1 to 255, while a_char contains the associated character code. An index was deliberately not defined for the column a_char.

Content:

id | a_char
. . . |
65 | 'A'
66 | 'B'
. . . |

importtable1, importtable2, exporttable

These three tables contain test data for importation and exportation of text files. A description of the tables as well as numerous possibilities for import and export by MySQL can be found in Chapter 10.


Chapter 6: An Introduction to SQL

Overview

This chapter provides an introduction to the database language Structured Query Language, or SQL for short. This language is used primarily to formulate instructions to a database system, including queries and commands to change or delete database objects. The most important of these commands are SELECT, INSERT, UPDATE, and DELETE, and these are the main attractions in this chapter, which aims to instruct in large part by presenting many examples.

Pointer

The following chapter, as a sort of continuation of the present one, provides a selection of recipes for solving everyday SQL problems, such as how to deal with character strings and with dates and times, and how to formulate subSELECT instructions (that is, SELECT commands whose results depend on another SELECT command).


Introduction

Prerequisites

All of the examples presented in this chapter are based on our example databases, and they use the database mylibrary, which was introduced in Chapter 5. If you wish to try out these databases for yourself, then you must download them to your own test environment.

Tip

Which database is involved in a particular example will be clear from the line USE databasename that appears at the beginning of the example or the beginning of the section in question.

The best way of testing simple SQL commands is to use the MySQL monitor mysql. This program can be launched under Unix/Linux with the command mysql. (Please note that you must terminate mysql commands with a semicolon. We shall not indicate the semicolon in this chapter, because it is not required by other client programs. That is, it is not part of the MySQL syntax. Additional information on the mysql monitor can be found in Chapter 4.)

When you get to the point of editing multiline SQL commands, you will find that the level of comfort and convenience afforded by mysql goes into a steep decline. A better alternative is to use one of the numerous MySQL interfaces (such as the MySQL Control Center). Such programs have the advantage that the results of a query are displayed in a reader-friendly tabular format.

Pointer

In this book we generally write SQL commands and key words in uppercase letters, reserving lowercase for the names of databases, tables, and columns. Sometimes, we use a mixed format (e.g., columnName). MySQL is largely case-insensitive. The exception is in the names of databases and tables, where case distinction is made. With names for such objects you must hew to the straight and narrow path of exactitude with regard to case. Details on naming rules can be found in Chapter 18.

DML, DDL, and DCL

SQL commands can be divided into three groups:

  • Data Manipulation Language (DML): SELECT, INSERT, UPDATE, and DELETE, and several additional commands, serve to read from the tables of a database and to store and update them. These commands form the centerpiece of this and the next chapter. Commands for controlling the transaction mechanism appear in Chapter 8.

  • Data Definition Language (DDL): With CREATE TABLE, ALTER TABLE, and their ilk the design (schema) of a database can be changed. Some such commands were used as examples in the previous chapter.

  • Data Control Language (DCL): GRANT, REVOKE, and several additional SQL commands help in setting the security mechansisms of MySQL. They will be presented in Chapter 10.

In Chapter 18 you will find an alpabetical reference to all SQL commands.


Simple Queries (SELECT)

Here is a simple database query: SELECT * FROM tablename. This query returns all data records of the table specified. The asterisk indicates that the query is to encompass all the columns of the table. (See Figure 6-1.)

Click To expand
Figure 6-1: First SQL experiments in MySQL monitor mysql
USE mylibrary
SELECT * FROM publishers
publID | publName
1 | Addison-Wesley
2 | Apress
3 | New Riders
4 | O'Reilly & Associates
5 | Hanser

Remark

SELECT can be used without reference to a database or to tables, for example, in the form SELECT 2*3, in which case SELECT returns its result in the form of a small table (one column, one row). This is used relatively often to determine the content of MySQL variables or functions (as in, say, SELECT NOW( ), to determine the current time).

Determining the Number of Data Records (Lines)

Perhaps you do not have a craving to view your data records in gruesome detail, but would like merely to determine how many records there are in one of your tables. For that, you can use the following query:

SELECT COUNT(publID) FROM publishers

COUNT(publID)

5

In this query you could specify, instead of publID, any column of the table (or * for all columns). In any case, MySQL optimizes the query and returns only the number of records, without actually reading them.

Column Restriction

Often, you are not interested in all the columns of a table. In such a case you must specify the columns explicitly (instead of using the asterisk).

SELECT publName FROM publishers

publName

Addison-Wesley

Apress

New Riders

O'Reilly & Associates

Hanser

Tip

If a query is going to return a large number of data records, you should get accustomed to the idea of specifying explicitly only those columns of interest (instead of taking the lazy person's route of simply typing an asterisk). The reason is that MySQL (unnecessary data extraction), the client program (memory usage), and the network (unnecessary data transfer) work significantly more efficiently if the data set to be processed is limited to the absolute minimum.

Limiting the Number of Resulting Records (LIMIT)

You can limit not only the number of columns in a query result, but also the number of data records. Imagine that your titles table contains the names of 100,000 books, but you would like to access only the first ten of these (for example, to display in an HTML document). To query the remaining 99,990 records would be a gross squandering of CPU time, memory, and network capacity. To avoid such a scenario, you can limit the number of records returned with LIMIT n. The following command returns two records from the titles table:

SELECT title FROM titles LIMIT 2

title

Client/Server Survival Guide

Definitive Guide to Excel VBA

To return the next two records, execute a new query, but this time with LIMIT offset, n. Here offset specifies the number of the record at which access to the table is to begin. (Warning: The enumeration of data records begins with 0. An offset of n skips the first n records and, since counting begins with 0, starts processing at record number n.)

SELECT title FROM titles LIMIT 2, 2

title

Linux

Web Application Development with PHP 4.0

Determing the Number of Records Suppressed by LIMIT (SQL_CALC_FOUND_ROWS, FOUND_ROWS)

When you execute a SELECT query with LIMIT, you obtain only a partial result. Particularly in the page-by-page display of data, it would often be helpful to know exactly how many records are available altogether.

Beginning with MySQL 4.0, you can use in a SELECT query the additional option SQL_CALC_FOUND_ROWS. Then, in a second query you can evaluate the SQL function FOUND_ROWS( ), which tells how many records the query would have returned without LIMIT.

The following query returns the three alphabetically first titles from the titles table. Then, with FOUND_ROWS it is determined how many records are available altogether:

USE mylibrary
SELECT SQL_CALC_FOUND_ROWS title FROM titles
ORDER BY title LIMIT 3

title

A Guide to the SQL Standard

Alltid den där Annette

Client/Server Survival Guide

SELECT FOUND_ROWS()

FOUND_ROWS()

26

The use of CALC_FOUND_ROWS and FOUND_ROWS is especially useful in complex queries in which a separate SELECT query for determing the number of records would be time intensive. Note, however, that the option CALC_FOUND_ROWS prevents certain kinds of optimization that MySQL carries out in LIMIT queries. Therefore, use CALC_FOUND_ROWS only when you then really wish to evaluate FOUND_ROWS.

Sorting Results (ORDER BY)

SELECT returns its results in no particular order. If you would like your list of results ordered, then you must request this explicitly via ORDER BY column. The following command returns an alphabetically ordered list of the authors in the mylibrary database.

SELECT authName FROM authors ORDER BY authName

authName

Bitsch Gerhard

Darween Hugh

Date Chris

DuBois Paul

Edwards E.

Garfinkel Simon

Gerken Till

Harkey D.

Holz Helmut

. . .

If you would like the reverse order, then you must add the key word DESC (for descending)to ORDER BY,as in, for example, ORDER BY authName DESC.

In sorting by character string, the order is determined by the character set chosen at the startup of MySQL. This is of particular importance when the character strings contain special characters outside of the ASCII character set. Information on selecting the character set for sorting can be found in Chapter 10.

Remark

In point of fact, the sequence of SELECT results, even without ORDER BY, is not random, but it depends rather on the order in which the records are stored in the table. However, you probably have no influence on this order. Therefore, do not expect that SELECT results will have a particular order. In the case of new tables, query results usually are returned ordered by increasing id number (if id is an AUTO_INCREMENT field). But as soon as records are altered or deleted, the ordering is ruined. Do not rely on it!
Tip

If you require your data records to be returned in a random order, then use ORDER BY RAND( ). For example, to choose two records from a table at random, use ORDER BY RAND( ) LIMIT 2.

Note, however, that ORDER BY RAND( ) LIMIT n without additional conditions is a very costly operation. For each record in the table, a random number must be generated. Then all the random numbers are sorted so that the first n records can be returned.

Selecting Data Records by Conditions (WHERE, HAVING)

Often, it is not all of the records in a table that are of interest, but only those that satisfy one or more conditions. Such conditionals are introduced with WHERE. In our first example, we wish to display the names of only those authors whose surname starts with one of the letters L through Z:

SELECT authName FROM authors WHERE authName>='L'

author

Orfali R.

Pohl Peter

Ratschiller Tobias

Reese G.

. . .

In our second example, we employ the operator LIKE to compare character strings. The query determines all authors whose names contain the sequence of letters er. With the operator LIKE, the character % serves as a placeholder for an arbitrary character string.

SELECT authName FROM authors WHERE authName LIKE '%er%'

author

Bitsch Gerhard

Gerken Till

Kofler Michael

Kramer David

Pohl Peter

Ratschiller Tobias

Schmitt Bernd

Yarger R.J.

Caution

Comparisons with LIKE can be very slow when they are applied to large tables. All the data records must be read and analyzed. It is impossible to use indexes to optimize such queries. A frequently used alternative to LIKE is the employment of afull-text index; see Chapter 7.

Comparisons with a large number of values can be carried out easily with IN:

SELECT authID, authName FROM authors
WHERE authID IN (2, 7, 12)
authID | authName
2 | Kramer David
7 | Gerken Till
12 | Yarger R.J.
Remark

Instead of formulating conditionals with WHERE, you could instead use HAVING. The WHERE conditionals are executed first, while HAVING conditionals are used only on intermediate results (returned by WHERE). The advantage of HAVING is that the conditions can also be applied to calculated fields (for example, to SUM(columnXy) in a GROUP BY query). An example appears in the section after next.

HAVING conditionals are less easily optimized for MySQL than WHERE conditionals, and they should be avoided if an equivalent WHERE is possible.
Remark

Please note that the conditional colname = NULL is not permitted. If you are searching for records that contain NULL,youmustwork with ISNULL(colname).


Queries with Related Tables

Up to now, all of our examples with SELECT have been applied to the search for records from a single table. However, with relational databases we are usually dealing with a number of related tables. Therefore, we are usually interested in applying a SELECT command to combine data from several tables. For this we will use the JOIN syntax.

JOINs Across Two Tables

A first attempt to create a list of all book titles (column title) together with their publishers (column publName) from the tables titles and publishers is a colossal failure:

USE mylibrary
SELECT title, publName FROM titles, publishers
title | publName
Client/Server Survival Guide | Addison-Wesley
Definitive Guide to Excel VBA | Addison-Wesley
Linux | Addison-Wesley
Web Application Development with PHP 4.0 | Addison-Wesley
Client/Server Survival Guide | Apress
Definitive Guide to Excel VBA | Apress
Linux | Apress
Web Application Development with PHP 4.0 | Apress
Client/Server Survival Guide | New Riders
Definitive Guide to Excel VBA | New Riders
Linux | New Riders
Web Application Development with PHP 4.0 | New Riders
. . . |

MySQL returns a list of all possible combinations of titles and publishers. In our relatively small database we can pick ourselves up from this minicatastrophe without too much damage. But imagine a database with 10,000 titles and 500 publishers, resulting in 5,000,000 combinations.

If queries spanning several tables are to return sensible results, precise information must be given as to how the data from the different tables are to be joined together. One possibility for formulating this connection is offered by WHERE. Since the linking field publID occurs in both tables, we must use the form table.column to indicate precisely which field is meant.

SELECT title, publName FROM titles, publishers
WHERE titles.publID = publishers.publID
title | publName
Linux, 5th ed. | Addison-Wesley
Definitive Guide to Excel VBA | Apress
Client/Server Survival Guide | Addison-Wesley
Web Application Development with PHP 4.0 | New Riders
MySQL | New Riders
MySQL & mSQL |
. . . |

There are quite a few additional ways of arriving at the same result. One variant consists in creating the table list with LEFT JOIN and then forming the linking connection with ON:

SELECT title, publName
  FROM titles LEFT JOIN publishers
    ON titles.publID = publishers.publID

Another variant makes use of the key word USING, in which the common linking field is specified. However, this variant assumes that the linking field (in this case publID) has the same name in both tables. That is, of course, not always the case:

SELECT title, publName
  FROM titles LEFT JOIN publishers
USING (publID)

JOINs Across Three or More Tables

Things become a bit more confusing when the query must examine data from more than two tables. The following query returns a list of all book titles with all their authors. (Books with several authors occur in this list with multiplicity.)

SELECT title, authName
  FROM titles, rel_title_author, authors
WHERE titles.titleID = rel_title_author.titleID
  AND authors.authID = rel_title_author.authID
ORDER BY title
title | author
A Guide to the SQL Standard | Date Chris
A Guide to the SQL Standard | Darween Hugh
Alltid den där Annette | Pohl Peter
Client/Server Survival Guide | Orfali R.
Client/Server Survival Guide | Harkey D.
Client/Server Survival Guide | Edwards E.
Definitive Guide to Excel VBA | Kofler Michael
Definitive Guide to Excel VBA | Kramer David
Excel 2000 programmieren | Kofler Michael
Jag saknar dig, jag saknar dig | Pohl Peter
LaTeX | Kopka Helmut
Linux für Internet und Intranet | Holz Helmut
Linux für Internet und Intranet | Schmitt Bernd
Linux für Internet und Intranet | Tikart Andreas
Linux, 5th ed. | Kofler Michael
Maple | Kofler Michael
Maple | Komma Michael
Maple | Bitsch Gerhard
. . . |

Our next example is somewhat more complex: Here we generate a list of publishers and their authors. (Thus the query determines which authors write for which publishers.) The connection between publishers and authors is made via the tables titles and rel_title_author,so that altogether, four tables are brought into play.

New in this query is the SQL key word DISTINCT. It has the effect that equivalent data records are output only once. Since there are authors in the mylibrary database who have written several books (for the same publisher), the simple joining of these tables would result in many combinations in which an author and publisher combination occurred with multiplicity.

SELECT DISTINCT publName, authName
  FROM publishers, titles, rel_title_author, authors
WHERE titles.titleID = rel_title_author.titleID
  AND authors.authID = rel_title_author.authID
  AND publishers.publID = titles.publID
ORDER BY publName, authName
publName | authName
Addison-Wesley | Bitsch Gerhard
Addison-Wesley | Darween Hugh
Addison-Wesley | Date Chris
Addison-Wesley | Edwards E.
Addison-Wesley | Harkey D.
. . . |
Apress | Kofler Michael
Apress | Kramer David
New Riders | DuBois Paul
New Riders | Gerken Till
New Riders | Ratschiller Tobias
. . . |

Syntax Variants

If you read about the details of FROM under the description of the SELECT syntax in the MySQL documentation, you may find your head spinning from the large number of (almost) identical variants. Here is a brief overview:

JOIN without condition (combination of all possibilities)

  1. FROM table1, table2

  2. FROM table1 JOIN table2

  3. FROM table1 CROSS JOIN table2

  4. FROM table1 INNER JOIN table2

  5. FROM table1 STRAIGHT_JOIN table2

In (1) through (4), MySQL tries on its own to find an optimal sequence for access to the data. Variant (5) is distinguished from the other four in that there, the order of data extraction from the tables is not optimized by MySQL. Option (5) is then to be recommended when you have doubts as to MySQL's ability to optimize its performance.

JOIN (with condition)

  1. FROM table1, table2 WHERE table1.xyID = table2.xyID

  2. FROM table1 LEFT [OUTER] JOIN table2 ON table1.xyID = table2.xyID

  3. FROM table1 LEFT [OUTER] JOIN table2 USING (xyID)

  4. FROM table1 NATURAL [LEFT [OUTER]] JOIN table2

  5. FROM table2 RIGHT [OUTER] JOIN table1 ON table1.xyID = table2.xyID

  6. FROM table2 RIGHT [OUTER] JOIN table1 USING (xyID)

With option (1), only those fields that are identical are considered in joining the two tables. With (2) and (4), on the other hand, for every record of the first (left) table a result record is generated, even when the linked field contains NULL. For example, if the titles table is used for table1, and the publishers table for table2, then LEFT JOIN will return even those titles for which no publisher name has been stored. (The key word OUTER is optional and has no effect on the function.)

Variants (5) and (6) correspond exactly to (2) and (3). Note, however, that table1 and table2 have been interchanged. RIGHT JOIN thus considers the query in a certain sense from the other direction. The MySQL documentation recommends not using RIGHT JOIN for improved compatibility with other databases, but instead using LEFT JOIN (with exchanged order of tables).

The variants with USING and NATURAL work only when the ID fields in both tables have the same name. In the case of NATURAL, like-named fields in both tables are used for joining. For NATURAL to work, the ID fields must have the same name and type. Furthermore, there must be no other like-named fields.

Warning

Please note that with LEFT JOIN the order of the tables is significant.

  • titles LEFT JOIN publishers returns titles that have no publisher, but no publishers that have not published a single title.

  • publishers LEFT JOIN titles returns, contrariwise, publishers that have yet to publish a title, but no titles that do not have an associated publisher.

In contrast to many other SQL dialects, MySQL recognizes no FULL JOIN,which would return all such combinations.


Uniting Query Results

Starting with MySQL 4.0, with UNION you can unite two or more SELECT queries. You thereby obtain a table of results in which the results of the individual queries are arranged one after another:

SELECT command UNION [ALL] SELECT command ...

The following example shows how the results of two SELECT commands on the same table are united. (Of course, the queries could also be with regard to separate tables, but then you would have to be sure that the tables had the same number of columns. Furthermore, the columns should exhibit the same data type. If that is not the case, then MySQL attempts to fit the data to the data type of the first SELECT command.)

USE mylibrary
SELECT * FROM authors WHERE authName LIKE 'b%'
UNION
SELECT * FROM authors WHERE authName LIKE 'g%'
authID | authName
22 | Bitsch Gerhard
26 | Garfinkel Simon
7 | Gerken Till

Normally, duplicate records are automatically eliminated from the result. Only when you use UNION ALL does the result contain the duplicates.

You can also place the individual SELECT terms in parentheses. Then, for each part of the query, as well as for the complete result, you can use LIMIT and ORDER as you wish. The following query first selects at most ten records from the tables tbl1 and tbl2 and unites them. From these (at most twenty) records, only the five most recent are displayed:

(SELECT * FROM tbl1 ORDER BY colA LIMIT 10)
UNION
(SELECT * FROM tbl2 ORDER BY colA LIMIT 10)
ORDER BY colTimestamp LIMIT 5
Remark

At present, UNION is the only set operator in MySQL. The key words MINUS and INTERSECT,available in many other database systems, are not availble in MySQL.


Grouped Queries, Aggregate Functions

The data records for our books, stored in the mylibrary database, are assigned to various categories. If you require information as to which categories contain which books, then you will find the following query helpful. (Warning: Books that have not been assigned to a category will not appear at all. If you wish to see all books, then you must execute FROM titles LEFT JOIN categories ON titles.catID = categories.catID.)

USE mylibrary
SELECT catName, title FROM titles, categories
WHERE titles.catID = categories.catID
ORDER BY catName, title
catName | title
Children's books | Alltid den där Annette
Children's books | Jag saknar dig, jag saknar dig
Computer books | LaTeX
Computer books | Linux, 5th ed.
Computer books | Maple
Databases | Client/Server Survival Guide
Databases | Visual Basic Datenbankprogrammierung
Programming | Definitive Guide to Excel VBA
Programming | Visual Basic
SQL | A Guide to the SQL Standard

If you wish to determine only how many books there are in each category, you can put the fingers of both hands to work and begin counting in the list just produced: two children's books, three computer books not more precisely categorized, two books on databases, and so on.

It has probably crossed your mind that there may be a more automated way of doing this: GROUP BY name creates in the resulting list a group for each member of the specified column. With GROUP BY catName in the above query a single row is made out of the two entries with catName='Children's books'. Which title is shown depends on the sort order, if any.

We see that with GROUP BY alone we do not accomplish much (unless we merely wanted to generate a list of all the categories, but there are easier ways of accomplishing that). However, in connection with GROUP BY, SQL supports aggregate functions.This means that in the column list at the beginning of a SELECT you can use functions like COUNT, SUM, MIN, and MAX. It is when these functions are brought into play that GROUP BY finally becomes a useful tool, as is demonstrated by the following example: We shall count (COUNT) how many entries are associated with each category.

New here is also the use of the key word AS, whereby the second column in the query is given the name nrOfItems. Without AS, the column would have the name 'COUNT(itemID)', which not only would be a bit confusing, but would increase the amount of typing each time the column was referenced (e.g., in ORDER BY ).

SELECT catName, COUNT(title) AS nrOfItems
FROM titles, categories
WHERE titles.catID = categories.catID
GROUP BY catname
ORDER BY catname
catName | nrOfItems
Computer books | 3
Databases | 2
Programming | 2
SQL | 1

If you would like a list of all categories, even those for which at present there are no books, then you must make the connection between the two tables with LEFT JOIN (as described in the previous section). This time, the resulting list will be sorted so that the categories with the most books are displayed first.

SELECT catName, COUNT(title) AS nrOfItems
FROM categories textbfLEFT JOIN titles ON titles.catid = categories.catid
GROUP BY catname

catName | nrOfItems
Computer books | 5
MySQL | 4
Programming | 3
Children's books | 3
PHP | 2
Databases | 2
Literature and fiction | 2
LaTeX, TeX | 1
SQL | 1
Object-oriented databases | 0
All books | 0
Relational Databases | 0
Perl | 0

GROUP BY for Several Columns

GROUP BY can also be used for several columns. The following query provides information about the numbers of book titles in various languages and the categories in which they appear. Note that with this query, all titles are lost that are not assigned a language or category.

SELECT COUNT(*), langName, catName
FROM titles, languages, categories
WHERE titles.catID = categories.catID
  AND titles.langID = languages.langID
GROUP BY titles.langID, titles.catID
COUNT(*) | langName | catName
1 | english | Computer books
1 | english | PHP
1 | english | SQL
2 | english | MySQL
5 | deutsch | Computer books
1 | deutsch | Databases
3 | deutsch | Programming
1 | deutsch | Relational Databases
1 | deutsch | PHP
1 | deutsch | Children's books
1 | deutsch | Literature and fiction
1 | deutsch | MySQL
1 | deutsch | LaTeX, TeX
2 | svensk | Children's books
3 | svensk | Literature and fiction
1 | norsk | Literature and fiction


Altering Data (INSERT, UPDATE, and DELETE)

Tip

MySQL novices frequently change or delete more records than intended in applying UPDATE and DELETE commands, by applying them to the entire table. If you launch mysql with the option --i-am-a-dummy, then the risk of accidental damage is reduced considerably: mysql refuses to execute UPDATE and DELETE commands without a WHERE.

Executing a Backup

Before you start playing around with your database, you should consider backing up individual tables or perhaps the whole database, so that after you make a mess of things, you can restore the database to its original pristine condition.

Creating Copies of a Table

The following instruction creates a new table with the name newtable and copies all the data records of table into the new table. The column definitions of the new table are identical to those of the old one, but there are some occasional differences. For example, the attribute AUTO_INCREMENT is lost. Moreover, in the new table, no indexes are created:

CREATE TABLE newtable SELECT * FROM table

Restoring Tables

With the given commands, first all data records of the original table table are deleted. Then the records that were saved into newtable are copied back into table. (The original AUTO_INCREMENT values remain untouched during the copying back and forth.)

DELETE FROM table
INSERT INTO table SELECT * FROM newtable

If youno longer require the backup data, you can simply delete newtable:

DROP TABLE newtable

Backing Up a Table

To the extent that you have access in MySQL to the file system (which depends on the security settings), you can back up a table very efficiently with BACKUP TABLE:

BACKUP TABLE table TO '/tmp/backups'

Instead of '/tmp/backups' you can specify an existing directory into which the files are to be copied. To recreate the table, execute RESTORE TABLE.

RESTORE TABLE table FROM '/tmp/backups'

BACKUP and RESTORE work only for MyISAM tables (not for InnoDB tables). Under Unix/Linux, the backup directory for the account under which MySQL is executed must be writable:

root# mkdir /tmp/backups
root# chown mysql /tmp/backups

Making a Backup of an Entire Database

With the program mysqldump (under Windows it is mysqldump.exe) you can create a backup file (in text format) of a complete database. Note, please, that you cannot execute mysqldump from within mysql, but rather, you must launch it as a freestanding program in a shell or command window. (The program will be described in detail in Chapter 10.)

user$ mysqldump -u loginame -p --opt dbname > backupfile
Enter password: xxx

Restoring a Database

There is no counterpart to mysqldump for reading in a saved database. Instead, you will have to rely on our old friend the SQL monitor mysql, where you give the backup file as input source. The database dbname must already exist.

user$ mysql -u loginname -p dbname < backupfile
Enter password: xxx

Of course, you can also recreate the database in interactive mode:

user$ mysql -u root -p
Enter password: xxx
Welcome to the MySQL monitor ...
mysql> CREATE DATABASE dbname; -- if dbname does not yet exist
mysql> USE dbname;
mysql> SOURCE backupfile;

Inserting Data Records (INSERT)

With INSERT you can add a new record to a table. After the name of the table there must appear first a list of the column names and then a list with the values to be inserted. (Columns with a default value, columns that can be NULL, and AUTO_INCREMENT columns do not have to be specified.)

In the following example, a new data record is saved in the table titles in the mylibrary database. Only two columns (title and year) will be specified. All remaining columns are taken care of by MySQL: A new AUTO_INCREMENT value is placed in titleID, and in the other columns, the requisite default value or NULL. (This process is allowed only because when the titles table was created, a default value was specifically provided for, and NULL was specified as a permissible value.)

USE mylibrary
INSERT INTO titles (title, year)
VALUES ('MySQL', 2001)

One may do without naming the columns if values for all columns (including default and AUTO_INCREMENT columns) are given and the order of the columns is followed exactly. In the case of titles, there are quite a few columns: titleID, title, subtitle, edition, publID, catID, langID, year, isbn, and comment.

For some columns, the value NULL is given. In most cases MySQL actually stores the value NULL. The only exception here is the column titleID, into which MySQL automatically places a new ID number. Instead of NULL, you can specify an explicit

ID number. However, this will result in an error if this number is already being used by another record. (There is one further column type in which MySQL places a value on its own if NULL is given: In TIMESTAMP columns, instead of NULL, the current date and time are stored.)

INSERT INTO titles
VALUES (NULL, 'deleteme', ' ', 1, NULL, NULL, NULL, 2001, NULL, NULL)

Another syntax variant enables several new records to be inserted with a single command:

INSERT INTO table (columnA, columnB, columnC)
VALUES ('a', 1, 2), ('b', 12, 13), ('c', 22, 33), ...

INSERT with Related Tables

If you are dealing with related tables, then normally, insertion of records is not accomplished with a single INSERT command. For example, to store a book record in the mylibrary database, new records must be stored in at least the tables titles and rel_title_author. If the new book was written by an author or authors whose names have not yet been stored in the database, or if it was published by a publisher that is currently unknown to the database, then new data records will also have to be added to the tables publishers and authors (see also Figure 5-2).

However, now we have the problem that not all the data for the insert command are known at the outset. To store a book published by a new publisher, the publID of the publisher must be given in the titles table. This is an AUTO_INCREMENT value, which is generated by MySQL only when the new publisher is stored in the database.

There must, then, be a way to access the last-generated AUTO_INCREMENT value. And indeed there is. The function for this is LAST_INSERT_ID( ), which returns the AUTO_INCREMENT value of the last INSERT command. This function has effect only within the current connection. That is, AUTO_INCREMENT values that may have arisen through the INSERT commands of other database users are ignored.

The following lines show how a book with a good deal of new information (three new authors, a new publisher) is stored. The book in question is the following: Randy Yarger, George Reese, Tim King: MySQL & mSQL. O'Reilly 1999.

INSERT INTO publishers (publName) VALUES ('O''Reilly & Associates')
SELECT LAST_INSERT_ID()
   4 <--- publID for the publisher
INSERT INTO authors (authName) VALUES (('Yarger R.')
SELECT LAST_INSERT_ID()
   12 <--- authorID for the first author
INSERT INTO author (author) VALUES (('Reese G.')
SELECT LAST_INSERT_ID()
   13 <--- authorID for the second author
INSERT INTO author (author) VALUES (('King T.'')
SELECT LAST_INSERT_ID()
   14 <--- authorID for the third author
INSERT INTO titles (title, publID, year)
VALUES ('MySQL & mSQL', 4, 1999)
SELECT LAST_INSERT_ID()
   9 <--- titleID for the book
INSERT INTO rel_title_author
VALUES (9, 12), (9, 13), (9,14)

In practice, you will not, of course, input these commands manually. That would be terribly boring, and the probability of error would be high. Rather, the insertions will be made in program code by way of a user interface. The code will concern itself not only with the evaluation and reevaluation of the ID values, but will ensure that authors and publishers who already appear in the database are not stored again by mistake.

Altering Data Records (UPDATE)

With UPDATE you can change individual fields of an existing database. The usual syntax of this command is the following:

UPDATE tablename
SET column1=value1, column2=value2, ...
WHERE columnID=n

Thus individual fields of a record specified by its ID value are changed. In the following example we change the title of the Linux book (titleID=1):

USE mylibrary
UPDATE titles SET title='Linux, 5th ed.' WHERE titleID=1

If UPDATE is used without WHERE, then the change is instituted for all data records (so beware!). The following command, for example, would change the publication date of all books to the year 2003:

UPDATE titles SET year=2003

With certain restrictions, calculations are permissible in UPDATE commands. Let us suppose that the price of the book was stored in the titles table. Then it would be easy to increase all prices by five percent:

UPDATE title SET price=price*1.05

Editing Data Records in Sorted Lists (UPDATE with ORDER BY and LIMIT)

If you want to change the first or last n data records, with the UPDATE command you can also specify ORDER BY and LIMIT. Such a possbility has existed only since MySQL 4.0. The following command sets the field mydata to zero for the first ten data records (in alphabetical order) of the table tablename:

UPDATE tablename SET mydata=0 ORDER BY name LIMIT 10

Updating with Linked Tables

Furthermore, since MySQL 4.0 you can process the data from several tables with UPDATE commands. The following example changes the column columnA of table1, where the new data come from table2.columnB. The link between the two fields is established via the common ID field table1ID (a genuine example of a multitable UPDATE command is given in Chapter 7:

UPDATE table1, table1
SET table1.columnA = table.columnB
WHERE table1.table1ID = table2.table1ID

Deleting Data Records (DELETE)

There can be little doubt that the syntactically simplest command of this section is DELETE. All records selected with WHERE are simply deleted. Here it is not required to specify the columns (indeed, it is not possible), since in any case, the entire data record is made to disappear without a trace. The following instruction deletes a single record from the titles table:

USE mylibrary
DELETE FROM titles WHERE titleID=8
Caution

If DELETE is executed without a WHERE condition, then all the records of the table will be deleted. There is no "undo" possibility.

The table itself with its definition of columns, indexes, etc., remains intact. If you wish to delete the table itself, then you must execute the command DROP TABLE.

Deleting Records from Linked Tables

In MySQL 3.23, the DELETE command can be used with reference to only a single table. Since MySQL 4.0, the following syntax is allowed:

DELETE t1, t2 FROM t1, t2, t3 WHERE condition1 AND condition2 ...

With this, all data records from tables t1 and t2 that satisfy the specified conditions are deleted. The conditions may contain arbitrary connections among the tables t1, t2, and t3.(To formulate it more generally, DELETE deletes records from only those tables that are given before FROM. In the conditions, records can be used from all tables that appear after FROM.)

The following example deletes from the titles table all book titles of the author Kofler Michael. To find the titles to be deleted, a link is created among the three tables tltles, rel_title_author, and authors:

USE mylibrary[1ex] DELETE titles FROM titles, rel_title_author, authors
WHERE titles.titleID=rel_title_author.titleID
  AND authors.authID=rel_title_author.authID
  AND authors.authName='Kofler Michael'

The following variant of the command given above deletes not only the titles, but Kofler Michael from the authors table, as well as all records from rel_tltle_author that create the link between the author Kofler Michael and his books (the difference here is that now all three tables are listed before FROM):

DELETE titles, rel_title_author, authors
  FROM titles, rel_title_author, authors
WHERE titles.titleID=rel_title_author.titleID
  AND authors.authID=rel_title_author.authID
   AND authors.authName='Kofler Michael'

The linkage between the tables can also be made via JOIN operators.

Deleting Data Records from Sorted Lists (DELETE with ORDER BY and LIMIT)

From version 4.0, ORDER BY can be used in DELETE commands. This makes sense only in combination with LIMIT. Now it is possible to delete the first or last n elements of a table based on an arbitrary sort criterion. The following command deletes the author with the highest authID number from the authors table:

DELETE FROM authors ORDER BY authID DESC LIMIT 1

Creating New Tables

You can store the result of a query directly in a new table. (In practice, this path is often followed in order to store the results of a complex query temporarily.) To do this, you simply specify the name of a new table with CREATE TABLE and insert an ordinary SELECT command at the end of the instruction. The following instruction copies all titles with catID=1 (category Computer books) into the new table computerbooks.

USE mylibrary
CREATE TABLE computerbooks
SELECT * FROM titles WHERE catID=1

From then on, the new table is available. With SELECT you can convince yourself that indeed the desired data are in the table:

SELECT title FROM computerbooks

title

Linux, 5th ed.

LaTeX

Mathematica

Maple

Practical UNIX & Internet security

When the time comes that you no longer have any need of the table computerbooks,you can easily delete it:

DROP TABLE computerbooks


Chapter 7: SQL Recipes

Overview

This chapter contains a collection of small SQL examples that go somewhat beyond the elementary level of the other examples in this chapter. The goal is to give you an idea of some of the possibilities that the world of SQL opens to you and to provide recipes for solving frequently occurring problems.

Most of these examples are based on the example databases introduced in chapter 5, and so you can easily try these commands out yourself. However, we will also introduce some techniques based on imaginary databases and tables.

Pointer

Examples of SQL commands appear in Chapter 5 on the subject of database design. Furthermore, many further examples are contained in the code in the examples of Chapters 12 through 17.


Character Strings

MySQL recognizes a large number of functions that allow for the processing of character strings in queries. The following examples introduce several of these functions. A reference for MySQL functions for processing character strings can be found in Chapter 18.

Basic Functions

Concatenating Character Strings

CONCAT(s1, s2, ... ) concatenates the given character strings into a single string. This function is useful for joining together several columns (for example, when you have stored first and last names in different columns and now wish to put them together):

SELECT CONCAT(firstname, ' ', lastname) FROM addresses

Extracting Parts of Character Strings

SUBSTRING(s, pos, n) returns n characters of the character string s starting at position pos. (The first character is addressed at position pos=1, not at pos=0.) The following command returns the first ten characters of all book titles form the mylibrary database:

USE mylibrary
SELECT SUBSTR(title, 1, 10) FROM titles

Those familiar with the syntax of the programming language Basic can use LEFT, RIGHT, and MID instead of SUBSTR to read strings conveniently from the beginning or from the end. We will offer an example of this later.

Determining the Length of a Character String

LENGTH(s) returns the number of characters in a character string.

Shortening Strings to a Particular Length

The function IF(a, b, c) evaluates expression a. If the result is (TRUE), the function returns b; otherwise, c.

The goal of the following example is to shorten book titles to the form "beginning of title, ... , end of title," so that the entire length of the resulting character string is at most thirty characters. Titles with thirty characters or fewer are unchanged:

SELECT IF(LENGTH(title)>30,
         CONCAT (LEFT(title, 20), ' ... ', RIGHT(title, 5)),
         title)
FROM titles AS shorttitle
shorttitle |
Linux für Internet u . . . | ranet
Mathematica |
Practical UNIX & Int . . . | urity
Visual Basic Datenba . . . | erung
. . . |

Pattern Matching

Pattern Matching with LIKE

The operator LIKE is suited for simple pattern matching. It supports two wild cards: _ for a single arbitrary character, and % for an arbitrary number (including 0) of characters. As with ordinary string comparison, LIKE does not distinguish between uppercase and lowercase. Here is an example:

SELECT 'MySQL' LIKE '%sql'
1

The following SELECT command returns all book titles that contain the character string SQL:

USE mylibrary
SELECT title FROM titles WHERE title LIKE '%SQL%'

If the character _ or % is itself to be used in the pattern to be matched, then it must be preceded by a backslash, as in '50%' LIKE '%\%', which returns 1. If you wish to use a character other than the backslash as the escape character, then you may specify such a character with ESCAPE: '50%' LIKE '%&%' ESCAPE '&', which also returns 1.

Remark

Please note that doing pattern matching in large tables is very slow. This holds for LIKE, andeven more so for REGEXP. In many cases it makes more sense to do a full-text search instead of pattern matching.

Pattern Matching with REGEXP

The command REGEXP offers many more possibilities for formulating a pattern, as does the equivalent command RLIKE. As with LIKE, case distinction is not made. Here we give simply a few syntatic examples. A table with the most important elements of REGEXP pattern matching appears in Chapter 18.

In the simplest case, expr REGEXP pattern tests whether the character string pattern is contained in expr:

SELECT 'abcabc' REGEXP 'abc', 'abcabc' REGEXP 'cb'
  1, 0

The search pattern does not have to describe the entire character string, but only a part. If you wish to encompass the entire character string, you must use ^ and $ in the search:

SELECT 'abc' REGEXP '^abc$', 'abcabc' REGEXP '^abc$'
  1, 0

Square brackets indicate a selection from among several characters, where a-c indicates the inclusive range from a to b. The operator + indicates that at least one of the characters must appear at least once in the search expression:

SELECT 'cde' REGEXP '[a-c]+', 'fgh' REGEXP '[a-c]+'
  1, 0

Parentheses indicate an entire character string, and curly braces indicate how many times the character string must appear in succession:

SELECT 'xabcabcz' REGEXP '(abc){2}', 'xabcyabcz' REGEXP '(abc){2}'

  1, 0

Note that REGEXP is successful when the the search pattern is found at any location within the character string.

Pointer

A large number of additional examples can be found in the appendix to the MySQL documentation:

http://www.mysql.com/doc/en/Regexp.html

Binary Character String Comparison

Character strings are generally compared without regard to case sensitivity. That is, 'a'='A' returns 1 as result. If you wish to execute a binary search, you must write BINARY in front of one of the operands. BINARY is a "cast" operator. That is, it changes the data type of an operand (thus changes a number or character string into a binary object). BINARY can be used for ordinary string comparison and also for pattern matching with LIKE and REGEXP:

SELECT 'a'='A', BINARY 'a' = 'A', 'a' = BINARY 'A'
  1, 0, 0
SELECT 'abcabc' REGEXP 'ABC', 'abcabc' REGEXP BINARY 'ABC'
  1, 0

Editing Character Strings

Naturally, you can also use string functions to alter text in MySQL tables. The following examples assume that there exists a table messages with the column author (VARCHAR(60)), which contains a name, an e-mail address, or some other text. The goal is to free this list of special characters and to make it anonymous. Therefore, everything except the first part of each name is to be expunged. Thus Peter Miller will become Peter, while paul.simon@garfunkel.net will be reduced to paul. This may be a somewhat unusual task, but you will see that the techniques introduced here are useful for many other applications.

Before having at it, you should make a copy of the column that is to be processed. Then if things go awry, you can restore the original state of the data:

ALTER TABLE messages ADD COLUMN origauthor VARCHAR(60)
UPDATE messages SET origauthor = author

In what follows, only author will be processed. The following command removes the double quote character ".For this, the function REPLACE is used:

UPDATE messages SET author = REPLACE(author, '"', '')

Next, each of the characters-,._@willbereplaced by a space:

UPDATE messages SET author = REPLACE(author, '-', ' ')
UPDATE messages SET author = REPLACE(author, '_', ' ')
UPDATE messages SET author = REPLACE(author, '@', ' ')
UPDATE messages SET author = REPLACE(author, ',', ' ')
UPDATE messages SET author = REPLACE(author, '.', ' ')

Now spaces will be searched for. If such a character is found, then apparently, the string consists of several parts (words). The UPDATE command truncates these strings to the first word. New here is the function LOCATE, which returns the position of a search pattern within a character string:

UPDATE messages SET author = LEFT(author, LOCATE(" ", author)-1)
WHERE LOCATE(" ", author) > 1

With SELECT one can check whether everything functioned as intended. (For privacy protection, here the original names and e-mail addresses have been replaced by xxx.)

SELECT author, origauthor FROM messages LIMIT 10
author | origauthor
Michael | Michael xxxxxxx
cjadner | cjadner@xxxxxx.xx
Frauke | "Frauke xxxxxxxxx"
Bernd | Bernd xxxxx, xxx@xxxxxxx.xx
jan | jan_xxxxx
test | test
Michael | Michael.xxxxxx@xxxxxx.xx
Thomas | Thomas xxxxxxxxx
Tobias | Tobias xxxx
sw410003 | sw410003@xxxxxx.xxx.xxx

And now the column origauthor can be deleted:

ALTER TABLE messages DROP origauthor


Date and Time

Basically, dates and times in SQL commands must be given as character strings, using a syntax that the MySQL server understands. The following three queries are equivalent, and work for both DATETIME and TIMESTAMP columns:

SELECT COUNT(*) FROM table
WHERE d BETWEEN '2001-05-16 08:34:07' AND '2002-02-11 00:15:44'
SELECT COUNT(*) FROM table
WHERE d BETWEEN '2001/05/16 08:34:07' AND '2002/02/11 00:15:44'
SELECT COUNT(*) FROM table
WHERE d BETWEEN '20010516083407' AND '20020211001544'

Note that BETWEEN a AND b means that the start and end times are included (that is, d>=a AND d<=b).

Determining the Number of Records by Day, Month, etc.

It happens quite often that you want to know how many records there are for a particular day or month. Of course, you can execute a number of queries with BETWEEN start AND end. But it is much simpler to format the data field with various functions and determine the number using GROUP BY and COUNT.

Our first example returns the number of votes by month for the year 2002 for the database bigvote. Month and year are determined by the MySQL functions MONTH and YEAR:

USE bigvote
SELECT COUNT(*), MONTH(ts) AS m
FROM votelanguage
WHERE YEAR(ts)=2002
GROUP BY m
COUNT(*) | m
152 | 1
227 | 2
. . . | . . .
58 | 12

If you want all results (not just for one year), then the expression for the grouping must contain both the year and the month. For this, you should use the DATE_FORMAT function (whose syntax is described in Chapter 18):

SELECT COUNT(*), DATE_FORMAT(ts, '%Y-%m') AS m_y
FROM votelanguage
GROUP by m_y
count(*) | m_y
214 | 2001-05
111 | 2001-06
226 | 2001-07
. . . | . . .
73 | 2003-01

Calculations with Date and Time

MySQL offers, alas, only a few SQL functions for dealing with date and time. Therefore, it is seldom possible to execute date and time calculations directly with SQL commands. Instead, it is frequently necessary to use a client program that first executes a SELECT query and then carries out the calculations.

As the following example demonstrates, there are exceptional cases in which success can be attained using SQL code. But for this you have to transform date and time into a purely numerical format. The basis for the following example is the table workingtimes,inwhich working hours can be recorded:

USE test
CREATE TABLE workingtimes (
  id INT AUTO_INCREMENT,
  begintime DATETIME,
 endtime DATETIME,
  PRIMARY KEY (id))
INSERT INTO workingtimes (begintime, endtime) VALUES
  ('2003-03-27 7:15', '2003-03-27 18:00'),
  ('2003-03-28 8:00', '2003-03-28 18:00'),
  ('2003-03-29 7:30', '2003-03-29 16:50'),
  ('2003-03-30 7:00', '2003-03-30 17:15')

The first attempt to calculate the working time for each workday fails miserably. The reason is that MySQL interprets the character strings for the subtraction endtime - begintime with time values simply as integers, for example, '2003-03-27 7:15:00' as 20030327071500. MySQL can subtract these numbers, of course, but the result is pure nonsense, since time is not recorded in the decimal system, with seconds and minutes in the range 0 to 59 instead of 0 to 99:

SELECT DATE_FORMAT(begintime, '%Y-%m-%d'), endtime - begintime
FROM workingtimes
DATE_FORMAT(begintime, '%Y-%m-%d') | endtime-begintime
2003-03-27 | 108500
2003-03-28 | 100000
2003-03-29 | 92000
2003-03-30 | 101500

With UNIX_TIMESTAMP you can transform time data into Unix timestamps. These are 32-bit integers that give the number of seconds that have elapsed since January 1, 1970. (Such a transformation thus makes sense only for times within the range 1970 to 2038.

For times before 1970, Unix timestamps are undefined, and after 2038, overflow kicks in.) After this transformation, the work time can be calculated in seconds:

SELECT DATE_FORMAT(begintime, '%Y-%m-%d') AS dt,
UNIX_TIMESTAMP(endtime) - UNIX_TIMESTAMP(begintime) AS s
FROM workingtimes
dt | s
2003-03-27 | 38700
2003-03-28 | 36000
2003-03-29 | 33600
2003-03-30 | 36900

To make these values in seconds comprehensible to human understanding, you can use the function SEC_TO_TIME. It returns a time value, where seconds have been calculated from midnight. (Times longer than twenty-four hours also appear in the form hhhh:mm:ss.Days are not calculated.)

SELECT DATE_FORMAT(begintime, '%Y-%m-%d') AS dt,
SEC_TO_TIME(UNIX_TIMESTAMP(endtime) - UNIX_TIMESTAMP (begintime)) AS t
FROM workingtimes
dt | t
2003-03-27 | 10:45:00
2003-03-28 | 10:00:00
2003-03-29 | 09:20:00
2003-03-30 | 10:15:00

The total working time can be easily determined with SUM, for example, for all days in March 2003:

SELECT SEC_TO_TIME(SUM(UNIX_TIMESTAMP(endtime)-UNIX_TIMESTAMP(begintime))) AS sumtime
FROM workingtimes
WHERE begintime>='2003-03-01 00:00:00' AND
begintime<='2003-03-31 23:59:59'

sumtime

40:20:00

In only a few cases do the SQL functions DATE_ADD and DATE_SUB help in carrying out such calculations. However, only a multiple of a particular interval can be added or subtracted, not an arbitrary time. (Details on the syntactic variants of DATE_ADD and DATE_SUB can be found in Chapter 18.) The following query returns the number of workdays with ten or more hours:

SELECT COUNT(*) FROM workingtimes
WHERE endtime >= DATE_ADD(begintime, INTERVAL 10 HOUR)

COUNT(*)

3

Peculiarities of TIMESTAMP

Avoiding Changes to TIMESTAMP

If you have a table with a TIMESTAMP column and execute an UPDATE on that table, then the content of the TIMESTAMP columns changes as well. (After all, the purpose of a TIMESTAMP column is to log the time of creation and the most recent change.)

It can happen, however, that in correcting a record you wish to avoid changing the TIMESTAMP value. For this, you can deal with the TIMESTAMP column explicitly with SET, returning the column to its previous value. The following example assumes that the TIMESTAMP column has the name ts:

UPDATE table SET data='new text', ts=ts WHERE id=123

When a large number of changes are in play, it is safer first to create a new column oldts of the previous TIMESTAMP values and copy all the TIMESTAMP values into it:

ALTER TABLE table ADD oldts DATETIME

UPDATE table SET ts=ts, oldts=ts

Now you can carry out all the changes you want without worrying about the ts column. When you are done, you simply restore the ts column to its original condition and delete the oldts column:

UPDATE table SET ts=oldts
ALTER TABLE table DROP oldts


Variables and Conditionals (IF, CASE)

The SQL dialect of MySQL contains few constructs that enable programming in the sense of a standard programming language. (This should change in MySQL 5.0, in which MySQL will allow the programming of custom functions with stored procedures.) Buteven with the few functions available now, we can do a bit of programming.

Variables

MySQL offers the possibility of storing simple values (scalars, but not lists, as, for example, the results of a SELECT command) in variables. These variables maintain their validity until the end of the MySQL connection. Variables are indicated by a prefixed "@" sign.

There are two syntax variants for variable assignment. If you are storing the result of a simple calculation, then SET is what you need, where either = or := may serve as the assignment operator:

SET @var1 := 3*4

More flexible than SET is SELECT, since with the latter, the assignment may now also contain the result of a query. With SELECT you must use := as the assignment operator. The following example returns from the table votelanguage from the questionnaire database test_vote (see Chapter 3) the time at which the last vote was cast:

USE test_vote SELECT @var2 := MAX(ts) FROM votelanguage

@var2 := MAX(ts)

20030115163054

You can also use SELECT to evaluate the contents of variables. Here case distinction is important (size matters):

SELECT @var1, @var2, @VAR2
@var1 | @var2 | @VAR2
20 | 20030115163054 | NULL

Of course, you can use variables in queries or in other SQL commands. Here, however, variables can be used only in expressions, not, for example, in table or column names:

SELECT * FROM votelanguage WHERE ts = @var2
id | choice | ts
13 | 5 | 20030115163054

With variables, you can also perform simple calculations. The following example shows how you can determine, for the bigvote database, the percentage of participants who voted a certain way:

USE bigvote SELECT @total := COUNT(*) FROM votelanguage

@total := COUNT(*)

3315

SELECT choice, COUNT(*) / @total * 100
FROM votelanguage
GROUP BY choice
choice | COUNT(*) / @total * 100
1 | 10.015082956259
2 | 11.070889894419
3 | 15.444947209653
4 | 49.502262443439
5 | 7.782805429864
6 | 6.184012066365

In practice, however, such calculations are seldom used. Instead, for programming of complex query or insertion operations you should use a programming language such as PHP or Perl that makes variables available and therefore makes the use of MySQL variables unnecessary (see also the introductory example in Chapter 3).

Pointer

MySQL also recognizes system variables on a global level and the level of the connection. The use of such variables is descibed briefly in Chapter 18.

IF Queries

With the function IF introduced earlier in this chapter, you can return one of two different results, depending on the evaluation of a condition:

IF(condition, result1, result2)

Of course, you can also nest IF queries:

IF(condition1, result1, IF(condition2, result2a, result2b))

If you wish to test whether a column contains NULL, you may use the function ISNULL (expr). In many cases, you can use IFNULL instead of a nesting of IF and ISNULL. The following expression returns expr2 if expr1 is NULL,and expr1 otherwise:

IFNULL(expr1, expr2)

corresponds to

IF(ISNULL(expr1), expr2, expr1)

CASE Branching

The CASE construct has two syntactic variants. The first of these returns result1 if expr=val1, result2 if expr=val2,and so on:

CASE expr
  WHEN val1 THEN result1
  WHEN val2 THEN result2
  ...
  ELSE resultn
END

With the second variant, it is not a condition that is evaluated, but an arbitrary number of conditions. The result is result1 if condition cond1 holds, result2 if condition cond2 holds, and so on:

CASE
  WHEN cond1 THEN result1
  WHEN cond2 THEN result2
  ...
ELSE resultn
END

The following example returns an alphabetical list of all English book titles (langID=1) from the mylibrary database. What is of interest here is that the articles a, an, and the are ignored in sorting. Thus A Programmer's Introduction will be found under the letter P, not A. To achieve this, sorting is not done directly by title, but using the CASE construct, which returns a truncated character string, with any article removed:

USE mylibrary
SELECT title FROM titles
WHERE langID=1
ORDER BY
  CASE
    WHEN LEFT(title,2)="A "
    THEN MID(title,3)
    WHEN LEFT(title,3)="An " THEN MID(title,4)
    WHEN LEFT(title,4)="The " THEN MID(title,5)
    ELSE title
END title

title

A Guide to the SQL Standard

MySQL

MySQL Cookbook

PHP and MySQL Web Development

Practical UNIX & Internet security

A Programmer's Introduction to PHP 4.0

In analogy to this example, you could do the same with articles in German (der, die, das), French (le, la, les), or Russian (just kidding, Russian has no articles). You could also filter out the www in web addresses or any other part of a character string.

Remark

Please observe that such search criteria significantly slow the execution of SQL queries. MySQL must first create a list of all titles and then sort them. It is not possible to use the existing index of the title column.


Copying Data from One Table to Another

Creating a New Table in Copying

With the command CREATE TABLE newtable SELECT ... FROM oldtable you can create a new table and insert into it all the records selected with a SELECT command. MySQL automatically generates all the necessary columns in newtable corresponding to the data selected by SELECT:

CREATE DATABASE newvote
USE newvote
CREATE TABLE votelanguage SELECT * FROM test_vote.votelanguage
    Query OK, 3328 rows affected (0.01 sec)
    Records: 3328 Duplicates: 0 Warnings: 0

If you now look at the table definition with SHOW CREATE TABLE oldtable or newtable, you will discover that in the process of copying, the index and AUTO_INCREMENT attribute for the column id have gone missing:

SHOW CREATE TABLE  test_vote.votelanguage -- (original)
    CREATE TABLE votelanguage (
          id         int(11) NOT NULL AUTO_INCREMENT,
          choice tinyint(4) NOT NULL default '0',
          ts         timestamp(14) NOT NULL,
          PRIMARY KEY   (id)
    ) TYPE=MyISAM
SHOW CREATE TABLE  newvote.votelanguage -- (new)
    CREATE TABLE votelanguage (
          id         int(11) NOT NULL default '0',
          choice tinyint(4) NOT NULL default '0',
          ts         timestamp(14) NOT NULL
    ) TYPE=MyISAM

The changes in table layout as a result of CREATE TABLE ... SELECT ... are burdensome in practice. Indexes are missing, column attributes disappear, VARCHAR tables become CHAR tables, and so on. Therefore, it is often better first to create the new table just like the old one with a separate CREATE TABLE command and then use INSERT INTO ... SELECT ... to fill it with data. The syntax for the CREATE TABLE command can be determined with SHOW CREATE TABLE oldtable:

CREATE TABLE votelanguage2 (
  id     INT NOT NULL AUTO_INCREMENT,
  choice TINYINT NOT NULL,
  ts     TIMESTAMP NOT NULL,
  PRIMARY KEY  (id)
) TYPE=MyISAM
INSERT INTO  votelanguage2  SELECT * FROM  test_vote.votelanguage

Starting with MySQL 4.1, you can create the new table more simply thus:

CREATE TABLE newtable LIKE oldtable
Pointer

Tables can also be copied and moved with the MySQL import and export commands. These commands are described in Chapter 10 in the discussion of backups.

Copying into Existing Tables

To copy data into an existing table, use the command INSERT INTO ... SELECT. You must make sure that the number of columns and their types match and that they are given in the correct order. If the source table has fewer columns than the target table, you can simply specify NULL or a fixed value for a column in the SELECT part, for example, SELECT NULL, id, choice, timest, 0, "abc":

INSERT INTO newvote.votelanguage
SELECT id, choice, ts FROM test_vote.votelanguage

The greatest source of problems with such commands is, yet again, AUTO_INCREMENT fields. If the source and target tables contain records with the same ID number, then an INSERT command results in an error (Duplicate entry xy for key ... ). There are several ways of getting around this problem:

  • You can use INSERT IGNORE INTO ... , which avoids the error message and leads to the affected records being simply not copied.

  • You can use REPLACE ... instead of INSERT INTO ... , which results in the affected records being overwritten on copying.

  • Youcan change the id column in copying so that no conflicts arise. (This variant is possible only if no other records are referred to or if the foreign key field is also changed accordingly.)

SELECT @maxid := MAX(id) FROM newvote.votelanguage;
INSERT INTO newvote.votelanguage
SELECT id+@maxid, choice, ts FROM test_vote.votelanguage


Pivot Tables

Pivot tables collect information from a data source and arrange it in groups. The simplest way to understand these tables is by way of an example: In the mylibrary table, for each book title there can be stored a publisher, a category, and a language. Suppose we wish to know how many titles there are for every combination of category and language. Pivot tables provide a simple statistical evaluation of table data.

Many database systems offer special OLAP commands for creating pivot tables. OLAP stands for online analytical processing and includes special methods for the management and evaluation of multidimensional data, where here "multidimensional" refers to the several characteristics by which the data can be grouped. OLAP-capable database systems are often called data warehouses.

Unfortunately, MySQL does not yet support such OLAP functions. (These are planned for version 5.0.) Therefore, you generally have to use external programs to create pivot tables. (Especially well suited for this task is Microsoft Excel; see also Figure 17-5.)

In simple cases, you can make use of SQL functions in MySQL. Of particular practical use is the function SUM in combination with IF. SUM allows for the return of the sum of a grouped column. With IF, the formation of the sum can be made dependent on a number of criteria. The following two examples show how to proceed.

The Pivot Tables titles, languages, categories

For our first example we use the mylibrary database. The goal is to return a table that tells how many titles there are in each combination of category and language (of course, only for those titles that have such information associated with them). The starting data look like this:

USE mylibrary
SELECT title, langname, catname
FROM titles, categories, languages
WHERE titles.catID = categories.catID
  AND titles.langID = languages.langID
title | langname | catname
A Guide to the SQL Standard | english | SQL
Practical UNIX & Internet security | english | Computer books
MySQL | english | MySQL
MySQL Cookbook | english | MySQL
PHP and MySQL Web Development | english | PHP
Visual Basic | deutsch | Programming
Excel 2000 programmieren | deutsch | Programming
LaTeX | deutsch | Computer books
Nennen wir ihn Anna | deutsch | Children's books
Mathematica | deutsch | Computer books
Maple | deutsch | Computer books
Visual Basic Datenbankprogrammierung | deutsch | Databases
MySQL | deutsch | MySQL
PHP 4 | deutsch | PHP
Mit LaTeX ins Web | deutsch | LaTeX, TeX
PostgreSQL | deutsch | Relational Databases
Anklage Vatermord | deutsch | Literature and fiction
Linux, 6th ed | deutsch | Computer books
Mathematica | deutsch | Computer books
Java | deutsch | Programming
Jag saknar dig, jag saknar dig | svensk | Children's books
Alltid den där Annette | svensk | Children's books
Kärleken | svensk | Literature and fiction
Comédia Infantil | svensk | Literature and fiction
Hunderna i Riga | svensk | Literature and fiction
Ute av verden | norsk | Literature and fiction

We will now group this list by category (GROUP BY ). For each category we will use SUM(IF(titles.langID=n, 1, 0)) to count the titles that appear in a particular language. COUNT(*) counts the titles in the category (independent of the category):

SELECT catName,
         SUM(IF(titles.langID=1, 1, 0)) AS english,
         SUM(IF(titles.langID=2, 1, 0)) AS deutsch,
         SUM(IF(titles.langID=3, 1, 0)) AS svensk,
         SUM(IF(titles.langID=4, 1, 0)) AS norsk,
         COUNT(*)
FROM titles, categories, languages
WHERE titles.catID = categories.catID
  AND titles.langID = languages.langID
GROUP BY catName
catName | english | deutsch | svensk | norsk | COUNT(*)
Children's books | 0 | 1 | 2 | 0 | 3
Computer books | 1 | 5 | 0 | 0 | 6
Databases | 0 | 1 | 0 | 0 | 1
LaTeX, TeX | 0 | 1 | 0 | 0 | 1
Literature and fiction | 0 | 1 | 3 | 1 | 5
MySQL | 2 | 1 | 0 | 0 | 3
PHP | 1 | 1 | 0 | 0 | 2
Programming | 0 | 3 | 0 | 0 | 3
Relational Databases | 0 | 1 | 0 | 0 | 1
SQL | 1 | 0 | 0 | 0 | 1

There is an obvious problem with this query in the SUM( ... ) terms. What would happen if the languages column contained not four, but twenty or thirty languages? In such cases you would have to write the SQL code for the query in a client program (e.g., in a PHP script): First you would obtain a list of all languages, then for each language, add a SUM( ... ) term to the SQL query.

The total numbers of titles for each language could be obtained thus:

SELECT langName, COUNT(*)
FROM titles, categories, languages
WHERE titles.catID = categories.catID
  AND titles.langID = languages.langID
GROUP BY langName
langName | COUNT(*)
deutsch | 15
english | 5
norsk | 1
svensk | 5

Pivot Table Query Results by Month

Our second example evaluates the table bigvote.votelanguage, with query results of a simple poll (see Chapter 3). The goal is to display for each month the percentages of votes, in order to determine a change in opinion or to detect an attempt at manipulating the results. The starting data, about 3300 records, look like this:

USE bigvote SELECT * FROM votelanguage LIMIT 5
id | choice | ts
1 | 4 | 20010516083407
2 | 3 | 20010516083407
3 | 5 | 20010516083407
4 | 4 | 20010516083407
5 | 2 | 20010516083407

It is easy to determine the number of participants in the poll by month:

SELECT DATE_FORMAT(ts, '%Y-%m') AS mnth, COUNT(*)
FROM votelanguage
GROUP BY mnth
textitmnth | textitCOUNT(*)
2001-05 | 214
2001-06 | 111
. . . | . . .
2002-12 | 58
2003-01 | 73

The number of votes for each programming language is determined with SUM(IF(... )):

SELECT DATE_FORMAT(ts, '%Y-%m') AS mnth,
       SUM(IF(choice=1, 1, 0))  AS c,
       SUM(IF(choice=2, 1, 0))  AS java,
       SUM(IF(choice=3, 1, 0))  AS perl,
       SUM(IF(choice=4, 1, 0))  AS php,
       SUM(IF(choice=5, 1, 0))  AS vb,
       SUM(IF(choice=6, 1, 0))  AS other,
       COUNT(*)
FROM votelanguage
GROUP BY mnth

mnth | c | java | perl | php | vb | other | COUNT(*)
2001-05 | 20 | 24 | 30 | 110 | 16 | 14 | 214
2001-06. . . | 6 | 13 | 15 | 64 | 7 | 6 | 111
2002-12 | 8 | 4 | 6 | 31 | 5 | 4 | 58
2003-01 | 7 | 14 | 13 | 24 | 10 | 5 | 73

If you now want to know the percentage distribution, you have to divide the SUM expressions by COUNT(*). The results are then made readable by multiplying by 100 and using the ROUND function:

SELECT DATE_FORMAT(ts, '%Y-%m') AS mnth,
       ROUND(SUM(IF(choice=1, 1, 0)) * 100 / COUNT(*), 1) AS c,
       ROUND(SUM(IF(choice=2, 1, 0)) * 100 / COUNT(*), 1) AS java,
       ROUND(SUM(IF(choice=3, 1, 0)) * 100 / COUNT(*), 1) AS perl,
       ROUND(SUM(IF(choice=4, 1, 0)) * 100 / COUNT(*), 1) AS php,
       ROUND(SUM(IF(choice=5, 1, 0)) * 100 / COUNT(*), 1) AS vb,
       ROUND(SUM(IF(choice=6, 1, 0)) * 100 / COUNT(*), 1) AS other
       FROM votelanguage
GROUP BY mnth

Determining the Number of Authors (mylibrary)

In the mylibrary database the association between the tables titles and authors is effected by means of the table rel_title_author. Yet it can happen that no author is associated with individual entries in the titles table. The following query determines the number of authors associated with each of the titles entries. The key point here is the use of LEFT JOIN (instead of simply enumerating the tables with FROM). This ensures that the result list also contains titles that do not have authors. The result is sorted by the number of authors, so that books with many authors are named first.

USE mylibrary
SELECT title, COUNT (authID) AS authCount
FROM titles LEFT JOIN rel_title_author USING(titleID)
GROUP BY titles.titleID
ORDER BY authCount DESC, title
title | authCount
Client/Server Survival Guide | 3
Maple | 3
MySQL & mSQL | 3
A Guide to the SQL Standard | 2
Definitive Guide to Excel VBA | 2
Practical UNIX & Internet security. . . | 2


SubSELECTs

Most database systems recognize subSELECTs, that is, some degree of nested SELECT queries. For example, to search for book titles whose publID refers to a nonexistent publisher, a query in the following form could be used:

USE mylibrary
SELECT * FROM titles
WHERE publID NOT IN
  (SELECT publID FROM publishers)

Such commands are not possible through and including version 4.0.n of MySQL, becoming available only with version 4.1. Until this version has trickled up to your Internet service provider, one must circumvent this restriction with well-designed JOIN operations or the use of temporary tables (even though such a modus operandi is not particularly intuitive). For example, the following command is equivalent to the one above:

SELECT titles.*
FROM titles LEFT JOIN publishers
ON titles.publID=publishers.publID
WHERE ISNULL(publishers.publID) AND NOT(ISNULL(titles.publID))

This section has two goals: to present application examples for the new subSELECT syntax in MySQL 4.1 (see also Appendix B) and to show that one can produce the equivalent of a subSELECT in MySQL 4.0 through a careful reformulation.

Tip

Additional examples using subSELECTs appear in the next section, which deals with searching for defective data records in tables.

Simple subSELECTs

Example 1: The following query returns all book titles whose publisher's name begins with "O." Then, a list of publID values for the key word IN is returned with a separate SELECT query.

USE mylibrary
SELECT title FROM titles WHERE publID IN
  (SELECT publID FROM publishers WHERE publName LIKE 'O%')

title

MySQL & mSQL

Practical UNIX & Internet security

MySQL Cookbook

Comédia Infantil

Hunderna i Riga

MySQL 4.1 recognizes an additional syntactic form, one that is less intuitive but that leads to the same result. In the following example, the second SELECT command, the one in parentheses, is executed for every book title. A book title returned by the first SELECT command is considered only if this second command has a nonempty result. The condition titles.publID = publishers.publID creates the relation between the two SELECT queries:

SELECT title FROM titles WHERE EXISTS
  (SELECT * FROM publishers WHERE
     titles.publID = publishers.publID AND
     publName LIKE 'O%')

In MySQL 4.0 and earlier versions, the same result can be attained by means of a simple JOIN operation. The similarity of the syntax with the EXISTS variant is unmistakable:

SELECT title FROM titles, publishers
WHERE titles.publID = publishers.publID AND publName LIKE 'O%'

Example 2: In this example we wish to return the oldest entry from the forum database bigforum. The time of this entry will be returned by the second query. The result of this query is used in the WHERE condition of the first query:

USE bigforum
SELECT msgID, timeSt FROM messages
WHERE timeSt = (SELECT MIN(timeSt) FROM messages)
msgID | timeSt
1 | 2000-09-30 00:31:02

This enquiry can also be effortlessly carried out with MySQL 4.0. To return the earliest entry, all entries are sorted according to date. Then LIMIT is used to select the first entry:

SELECT msgID, timeSt FROM messages ORDER BY timeSt LIMIT 1

The Use of Temporary Tables

It is not always possible to replace subSELECTs by JOIN operations, for example, when the resulting list of records is to be sorted in a way different from that used in the selection query.

In the following example we wish to create an alphabetic list of the contributors of the previous ten listings in the bigforum database. In MySQL 4.1, the query looks like this:

USE bigforum SELECT DISTINCT userName FROM users
WHERE userID IN
  (SELECT userID FROM messages
   ORDER BY timest DESC
   LIMIT 10)
ORDER BY username

userName

Cees

Graham

matadordb

. . .

With a direct linkage between users and messages you can achieve a similar result, but then one cannot have either a sort by userName or the elimination of duplicates with DISTINCT. You must sort according to timest, so that LIMIT has the desired effect, namely, the selection of the first entry:

SELECT userName, messages.timest FROM users, messages
WHERE users.userID = messages.userID
ORDER BY messages.timest DESC LIMIT 10
username | timest
test | 20030207102132
Cees | 20030124154825
matadordb | 20030124132214
Michael | 20030124092312
Witzer | 20030124052830
Graham | 20030123211616
Volkan | 20030123155707
Michael | 20030123083245
Peter | 20030122175147
Paul | 20030121220412

The only possible solution without subSELECTs consists in using a temporary table to store intermediate results. As table type you would use HEAP. This has the effect of keeping the temporary table in RAM (not on the hard drive, that is), which speeds up the operation considerably:

CREATE TEMPORARY TABLE tmp TYPE=HEAP
SELECT userID, timest FROM messages
ORDER BY timest DESC
LIMIT 10

The contents of the temporary table are now linked with users:

SELECT DISTINCT userName FROM users, tmp
WHERE users.userID = tmp.UserID
ORDER BY username

username

Cees Graham

matadordb

. . .

The temporary table can now be deleted:

DROP TABLE tmp


Ensuring the Integrity of Data

During the creation of a database project (database design, development of program code, and so on), it often happens that test data sets are incompletely deleted and some references to tables lead nowhere. Of course, programming errors can also lead to such problems in the actual operation of a database system, which is always an indication that somewhere or other there is an error in the code.

Whatever the cause, one of the burdensome tasks of database administration is to manage the integrity of the data and correct errors as they are discovered. In the following, you will find some tips for such error management. All the examples refer to the database mylibrary.

Searching for Titles Without Authors

Every title in the mylibrary database is required to have at least one author. The relationship between the tables titles and authors is via the table rel_title_author. To determine the number of authors for each title, the following query evaluates the tables titles and rel_title_author. Crucial here is the use of LEFT JOIN (instead of simply enumerating titles and rel_title_author with FROM): This ensures that the resulting list will contain all titles that have no associated authors. The result is sorted according to the number of authors, with titles with more authors coming before those with fewer authors:

USE mylibrary
SELECT title, COUNT(authID) AS authCount
FROM titles LEFT JOIN rel_title_author USING(titleID)
GROUP BY titles.titleID
ORDER BY authCount DESC, title
title | authCount
Client/Server Survival Guide | 3
Maple | 3
MySQL & mSQL | 3
A Guide to the SQL Standard | 2
Definitive Guide to Excel VBA | 2
Practical UNIX & Internet security | 2
. . . |

It is then just a small step from the above query to finding the titles for which no author has been stored. In this example there is only one title that appears to have slipped into the database during a test run:

USE mylibrary SELECT title, COUNT(authID) AS authCount
FROM titles LEFT JOIN rel_title_author USING(titleID)
GROUP BY titles.titleID
HAVING authCount=0
ORDER BY title

title

dummy

Remark

Be sure to note that we have used HAVING here (and not WHERE!). WHERE is only for conditions that affect the original data, while authCount appeared as the result of the evaluation of an aggregation function.

Starting with MySQL 4.1 you can retrieve such titles with subSELECT queries. The following query tests each title in titles to see whether there is an entry in rel_title_author with a corresponding titleID number. Only the titles that do not have such an entry are displayed (due to NOT EXISTS):

SELECT title FROM titles WHERE NOT EXISTS
  (SELECT * FROM rel_title_author
    WHERE titles.titleID = rel_title_author.titleID)

Searching for Invalid Publisher Links: Invalid Records in 1: n Relations

In an ideal world, all the records in the titles table should refer with publID to a publisher, or else the relevant field should contain NULL. But what happens if a publisher is inadvertently deleted? Then there are titles records with publIDs that refer to a no longer existing publisher. (Similar problems can arise with other relations between other tables, with the titles table and the catID and langID fields. The following query thus does not find all titles records with invalid cross references, but only those whose publID is screwed up. Thus we are merely giving an example of how to find such records.)

To simulate the problem, we shall insert a defective title into the titles table. The command assumes that there is no publisher with publID=9999:

INSERT INTO titles (title, publID) VALUES ('deleteme', 9999)

The following query links the tables titles and publishers via the linking field publID. With LEFT JOIN, all the titles are found that lead to no result via this link. This can be for one of two reasons: Either titles.publID has the value NULL (which is permitted), or for titles.publID there is no corresponding entry in the publishers table (and an error has been found). The two ISNULL conditions are necessary so that in the end, only those records are displayed that were found for the second reason.

The query finds in titles only a single record (which was added expressly for this purpose):

SELECT title, titles.publID, publishers.publID
FROM titles LEFT JOIN publishers
ON titles.publID=publishers.publID
WHERE ISNULL(publishers.publID)
  AND NOT(ISNULL(titles.publID))
title | publID | publID
deleteme | 9999 | NULL

Starting with MySQL 4.1, this query can be formulated as a subSELECT command:

SELECT * FROM titles
WHERE publID NOT IN (SELECT publID FROM publishers)

There are two ways to correct such errors: You can simply delete any invalid titles (which in this case is the called-for solution), or you can associate the title with a publisher (which you will have to create if the publisher does not already appear in the list of publishers).

Searching for Invalid Authors and Titles (n : m Relation)

Somewhat more complicated is the search for n : m relations (such as in the title/author relationship). Now there are three sources of error:

  • Titles that refer via rel_title_author to nonexistent authors;

  • Authors that refer via rel_title_author to nonexistent titles;

  • Entries in rel_title_author for which both titleID and authID refer to invalid records. (If the search for illegal records is based on existing authors or titles, then this third source of errors remains unexposed.)

To simulate the error situation, three invalid records will be inserted into rel_title_author, covering all three error sources. The following instruction assumes that there exists an author with authID=1 and a title with titleID=1, but no author or title with ID number 9999:

INSERT INTO rel_title_author (titleID, authID)
VALUES (1,9999), (9999,1), (9999, 9999)

The following query refers to the first error (title with author IDs that do not exist in the database). It is interesting in this query that the link between titles and rel_title_author is effected with an ordinary JOIN, while that between rel_title_author and authors is done with LEFT JOIN (otherwise, the missing authors would not be included and thus would not be not found). The result is not very surprising: The title Linux with titleID=1 refers to an author with authID=9999, which doesn't exist.

SELECT title, rel_title_author.titleID, rel_title_author.authID
FROM titles, rel_title_author
LEFT JOIN authors
ON rel_title_author.authID=authors.authID
WHERE titles.titleID=rel_title_author.titleID
  AND ISNULL(authors.authID)
title | titleID | authID
Linux, 5th ed. | 1 | 9999

The symmetric command for the second error case looks like this:

SELECT authName, rel_title_author.authID, rel_title_author.titleID
FROM authors, rel_title_author
LEFT JOIN titles
ON rel_title_author.titleID=titles.titleID
WHERE authors.authID=rel_title_author.authID
  AND ISNULL(titles.titleID)
authName | authID | titleID
Kofler Michael | 1 | 9999

The query for the third error case consists of two LEFT JOINs, one from rel_title_author to authors, and the other from rel_title_author to titles.

SELECT rel_title_author.authID, rel_title_author.titleID
FROM rel_title_author
LEFT JOIN titles
  ON rel_title_author.titleID=titles.titleID
LEFT JOIN authors
  ON rel_title_author.authID=authors.authID
WHERE ISNULL(titles.titleID) AND ISNULL(authors.authID)
authID | titleID
9999 | 9999

Instead of the three queries just formulated, we can get by with only one. For this to work, all the data in all three tables must be considered. Furthermore, in the conditional we will need to use AND and OR. (In practice, it is often more useful to split the query into three parts, because then one can tailor the response to the different types of error.)

SELECT rel_title_author.authID, rel_title_author.titleID,
authName, title
FROM rel_title_author
LEFT JOIN titles
  ON rel_title_author.titleID=titles.titleID
LEFT JOIN authors
  ON rel_title_author.authID=authors.authID
WHERE ISNULL(titles.titleID) OR ISNULL(authors.authID)
authID | titleID | authName | title
9999 | 1 | NULL | Linux, 5th ed.
1 | 9999 | Kofler Michael | NULL
9999 | 9999 | NULL | NULL

One of the most important results of this example should be your realization of the universal applicability of JOIN operations. In MySQL 4.1, you can simplify your life by formulating the query as a subSELECT command:

SELECT titleID, authID FROM rel_title_author
WHERE authID NOT IN (SELECT authID FROM authors) OR
titleID NOT IN (SELECT titleID FROM titles)
authID | titleID
9999 | 1
1 | 9999
9999 | 9999

Correcting Data (n : m relations)

Of course, it helps in correcting data to be able to determine which data are corrupt. It would be even better if one could correct errors automatically as soon as they were found. Alas, as a rule, errors must be corrected manually. For example, if a book title refers to a nonexistent author because of an error in rel_title_author, then it will not do simply to delete this entry. It must first be determined whether the author list for the book is, in fact, in error. It may be that a new author (or only a new reference in rel_title_author to an existing author) has to be incorporated into the database.

Even with the best SQL command you cannot automate such tasks. However, SQL is a good aid in searching through the database for errors and listing them in a readable format as a basis for manual correction to follow.

When the problem is only the syntactic correctness of data (and you have no possibility of tracking the error to its source and then manually recreating the data), you can, of course, simply delete erroneous data (for example, all records in rel_title_author that refer to a nonexistent author or title).

Since MySQL 3.23.n and all earlier versions do not, unfortunately, offer the possibility of accessing data in several tables with DELETE commands (no subSELECTs), you must carry out such tasks in an external programming language (in Perl, for example). There you first determine with SELECT the list of erroneous data records and then delete them with DELETE commands. An example of this technique can be found in Chapter 14.

In MySQL 4.0, the entire process is made easier in that now, data from several tables can be considered. The following command deletes all invalid records from the table rel_title_author:

DELETE  rel_title_author
FROM  rel_title_author
LEFT JOIN  titles
     ON  rel_title_author.titleID=titles.titleID
LEFT JOIN  authors
      ON  rel_title_author.authID=authors.authID
WHERE ISNULL(titles.titleID)  OR ISNULL(authors.authID)

In MySQL 4.1, it is possible to combine DELETE with SELECT:

DELETE FROM rel_title_author
WHERE authID NOT IN (SELECT authID FROM authors) OR
      titleID NOT IN (SELECT titleID FROM titles)


Tracking Down Duplicates

In database management, the problem repeatedly arises that data records are stored inadvertently in duplicate. As the following example involving the table authors and the database mylibrary shows, such records are easy to find. To this end, we form groups of identical author names and then display those groups with more than one member:

USE mylibrary
SELECT authName, COUNT(*) AS cnt
FROM authors
GROUP BY authName
HAVING cnt>1

Ina second step, one could obtain all the authID numbers for these records. Then duplicate entries can be removed, and you should replace in the table rel_title_author the second authID number by the first, and then delete the second entry from the table authors.

For example, suppose that due to an input error the author Gary Cornell appears twice:

authID | authName
123 | Cornell Gary
758 | Cornell Gary

In the titles table there are two different books by Gary. The first of these, The Sex Life of Unix, is associated with authID=123, while the second book, More Sex Life of Unix, has authID=758.Gary has two authIDs!

To alleviate this problem, we decide to use only authID=123, and so we change all rel_title_author entries with 758 to 123:

UPDATE rel_title_authors SET authID=123 WHERE authID=758

Now all of Gary's books have authID=123. Finally, we delete the obsolete entry in the authors table:

DELETE FROM authors WHERE authID=758

In the titles table it is permitted for several books to have the same title. Therefore, several columns must be considered:

SELECT title, subtitle, edition, COUNT(*)AS cnt
FROM titles
GROUP BY title, subtitle, edition
HAVING cnt>1

It is more difficult to track down duplicates that involve typographical errors. With English words one can use SOUNDEX. This function returns a character string that corresponds to the pronunciation of the text. However, the result must be manually examined. SOUNDEX does not find all similar records, and it considers as duplicates names that are really different. The following query uses MIN(authName) to return the alphabetically first of the similar-sounding names:

SELECT SOUNDEX(authName) AS snd,
       COUNT(*)AS cnt,
       MIN(authName) AS firstname
FROM authors
GROUP BY snd
HAVING cnt>1


Bringing a Table into Normal Form

In Chapter 5 we described in detail the first three normal forms, which are considered the starting point for good database design. In the best of all possible worlds, all design issues are dealt with at the outset, but in the world as we know it, it frequently happens that one comes to grips with these rules after it is too late.

But what does that mean, "too late"? The example of this section shows how you can move redundant information into a new table, thereby creating the third normal form after the fact.

The starting point for our example is the table messages, which contains a field author with the names of the contributors to an on-line discussion forum. Since many authors contribute multiple messages, their names appear repeatedly. This shortcoming will be corrected with the new table users, which is to contain a list of all the authors' names. In messages, the author column will be replaced by a userID column.

The first step is to create the new table users:

CREATE TABLE users (userID INT NOT NULL AUTO_INCREMENT,
                    username VARCHAR(60) NOT NULL,
                    PRIMARY KEY (userID))

With the command INSERT INTO ... SELECT introduced earlier in this chapter, all names can be inserted into the new users table. SELECT must be executed with the option DISTINCT, so that each name is inserted only once. For the userID column, the value NULL is passed. This leads to MySQL placing AUTO_INCREMENT numbers in the column automatically:

INSERT INTO users SELECT DISTINCT NULL, author FROM messages

Now the original table messages must be expanded to contain the new userID column:

ALTER TABLE messages ADD userID INT

Next, the userID column should be filled with data in such a way that the numbers refer to the corresponding entries in the user columnn. Note, please, that the UPDATE command in this form is valid only starting with version 4.0. (MySQL 3.23 offers no support for UPDATE commands in which data from more than one table are to be processed.) Although UPDATE refers to both messages and users, it is only the former that is changed. (Only messages.userID is newly set by SET.)

UPDATE messages, users
SET messages.userID = users.userID
WHERE messages.author = users.username

Abrief check shows that everything worked as planned:

SELECT messages.author, users.userName, users.userID
FROM messages, users
WHERE messages.userID = users.userID LIMIT 5
author | userName | userID
boehnke | boehnke | 9
Michael | Michael | 10
cjander | cjander | 11
Frauke | Frauke | 12
Bernd | Bernd | 13

Now the column messages.author can be deleted; the information contained therein is recoverable via messages.userID and the new users table:

ALTER TABLE messages DROP author


Processing the First or Last n Records

All the examples of this section use the test database bigforum, a large version of myforum.

Searching Data (SELECT)

The question repeatedly arises how to construct a list of the "top ten," or, better, the top n, which means the n first, the best, the worst, the oldest, or the most pleasing records in a data set. This question is easily answered: Use a SELECT command in combination with ORDER BY and LIMIT.

With ORDER BY you have to give a sort criterion, for example, a date. If you want not the first, but the last results, then reverse the sort order with DESC. (For example, ORDER BY date DESC, and LIMIT restricts the number of results to a particular value.)

The following query returns the last five entries in the bigforum discussion groups. With LEFT, the subject text is reduced to twenty characters, to make the result easier to read:

USE bigforum SELECT LEFT(subject, 20) AS subj, forumID, timest
FROM messages ORDER BY timest DESC LIMIT 5
subj | forumID | timest
Re: Run-time error ' | 1001 | 20030124154825
Re: Re: Suche Erfahr | 1 | 20030124132214
Re: Full text search | 1006 | 20030124092312
Re: MySQL on ASP wit | 1006 | 20030124052830
Full text searches | 1006 | 20030123211616

Our second example searches not for arbitrary messages, but for those that start a new thread (rootID=0), and only for the forum 1006 (MySQL, English):

SELECT LEFT(subject, 20) AS subj, forumID, timest FROM messages
WHERE rootID=0 AND forumID=1006
ORDER BY timest DESC LIMIT 5
subj | forumID | timest
Full text searches | 1006 | 20030123211616
MySQL and ASP | 1006 | 20021021223216
where do I save this | 1006 | 20021016013629
samples do not work | 1006 | 20021014151020
ASP and MySQL | 1006 | 20020730224220

In our next example, the five longest threads are found (that is, the threads that have the largest number of contributions). For this, COUNT and GROUP BY are used to return to number of messages that have the same rootID number. (The rootID number refers to the first message of the thread.) We must exclude rootID=0, since that indicates the beginning of a thread:

SELECT COUNT(*)AS answerCount, rootID FROM messages
WHERE rootID<>0 GROUP BY rootID ORDER BY answerCount DESC LIMIT 5
answerCount | rootID
11 | 767
9 | 1392
8 | 1134
7 | 495
7 | 748

Unfortunately, with this query it was impossible to display the subject. The reason is that GROUP BY groups all the messages of a thread, and with COUNT the number of messages can be determined. But there is no string function that distinguishes the first of several subject texts. (With numerical values, one could use MIN or MAX.) You must therefore work either with temporary tables or manually execute an additional query:

SELECT subject FROM messages WHERE msgID IN (767, 1392 ... )

Changing Records (UPDATE and DELETE)

Just as with SELECT you can select the first or last n records using ORDER BY and LIMIT n,you can as well change these records with UPDATE and DELETE.

The first of our two examples is rather trivial: To save space, UPDATE replaces the text of the oldest 100 messages with the bilingual notification message no longer available/Nachricht steht nicht mehr zur Verfügung. Important here is the instruction timest=timest, which ensures that timest is left unchanged. (Otherwise, the oldest 100 messages would suddenly seemingly become the newest.)

UPDATE messages
SET msgText="message no ... / Nachricht ... ", timest=timest
ORDER BY timest LIMIT 100

It is even easier to delete the last 100 messages:

DELETE messages ORDER BY timest LIMIT 100

Both commands are easily understood, but they have the disadvantage that they destroy the threads. It can also happen that the start message of a thread is overwritten or deleted, while (newer) replies remain. That is not, of course, what we had in mind.

Delete All Threads Except the Last 500

The following commands delete complete threads except for the most recent 500. Unfortunately, the process is somewhat complex. The first command returns the time

at which the oldest of the five hundred most recent threads was begun. This time is stored in the user variable @oldtime:

SELECT @oldtime := timest FROM messages
WHERE rootID=0 ORDER BY timest DESC LIMIT 499,1

@oldtime := timest

20030911093909

The next command tests how many threads exist that are even older. This information is not necessary for further processing, but it constitutes a sort of plausibility control:

SELECT COUNT (*) FROM messages WHERE timest<@oldtime AND rootID=0

COUNT(*)

117

Nowwe need to return a list of msgID numbers of the start messages of the threads that are to be deleted. This is written to a temporary table msgIDs using CREATE ... SELECT. This table will be stored in RAM for reasons of efficiency (TYPE=HEAP):

CREATE TEMPORARY TABLE msgIDs TYPE=HEAP
SELECT msgID FROM messages WHERE timest<@oldtime AND rootID=0

The table msgIDs is now used to delete the old messages, in fact, both the start messages (messages.msgID = msgIDs.msgID) and the replies (messages.rootID = msgIDs.msgID):

DELETE messages FROM messages, msgIDs
WHERE messages.rootID=msgIDs.msgID OR messages.msgID=msgIDs.msgID

Now the temporary table msgIDs can be deleted:

DROP TABLE msgIDs


Selecting Random Records

It is sometimes desirable to select data records at random, for example, to display images or advertisements on a web site.

Selection without a random Column

The easiest procedure is to obtain a random selection by selecting all records with SELECT, but executing ORDER BY with the addition of RAND( ). This will yield a list sorted by random numbers. Then LIMIT can be used to select the first record.

The following example is based on a copy of the titles table from the mylibrary database. With two queries, two (possibly) different (randomly selected) titles are selected from the copy titlescopy:

USE mylibrary
CREATE TABLE titlescopy SELECT * FROM titles SELECT titleID, title FROM titlescopy
ORDER BY RAND() LIMIT 1
titleID | title
59 | PHP and MySQL Web Development
SELECT titleID, title FROM titlescopy
ORDER BY RAND() LIMIT 1
titleID | title
2 | Definitive Guide to Excel VBA

If you wish to select two or more records randomly with only one SELECT query, then specify LIMIT n instead of LIMIT 1.

Selection with a random Column

The process just described has an obvious drawback: With large tables, it is extremely inefficient. All the records must be read into active memory, and there sorted by random numbers, all in order to select a single record.

For large tables it is therefore a good idea to provide the table with a column of random numbers at the time that the table is declared. Of course, the random column can also be created after the fact, as the following lines prove: The first ALTER TABLE command adds the column, while the second equips the column with an index. UPDATE then stores a random value in the new column for each existing record. (When a new record is added, you must then supply a new random number for the random column. Unfortunately, at present, MySQL does not allow a function such as RAND to be specified as a default value.)

ALTER TABLE titlescopy ADD random DOUBLE
ALTER TABLE titlescopy ADD INDEX (random)
UPDATE titlescopy SET random = RAND()

Now, to select a random record, use the following command:

SELECT titleID, title FROM titlescopy
WHERE random > RAND() ORDER BY random LIMIT 1

At first glance, this does not seem any more efficient than the previous example. In fact, however, MySQL now has selected only a single record from the table that satisfies the condition random > RAND(). Since an index was constructed for the random column, access is very efficient.

Do not forget ORDER BY ! Otherwise, MySQL chooses any old record that satisfies random > RAND(), and not necessarily the one whose value is closest to the random number. The selection of a record would then no longer be purely random, but would depend on how the records are stored in the table.

Of course, the variant with the random column is not without its own drawbacks:

  • More space on the hard drive is required to store the table (additional column plus index).

  • Every change in the table is slower than previously (index for the random column).

  • It is possible to select only a single random record. (If you use LIMIT 5, then you obtain five records to be sure, but it is always the same package of five records.)

  • On inserting new records, you must always assign a random number to the random column.

Selection on the Basis of an id Column

Perhaps both of the solutions that we have offered seem unnecessarily complicated. We have only to read the nth of nmax records, where nmax can be easily obtained with SELECT COUNT(*) FROM table. Unfortunately, there is no SQL command to read the nth data record. MySQL decides the order in which the records are stored (for example, the table driver), and this cannot be controlled from outside.

One solution would be to use an AUTO_INCREMENT column. If the table contains 1000 records, then the id column would have to contain the values from 1 to 1000, and the selection would be possible with WHERE id = CEILING(RAND()*1000). The problem, however, is that you cannot rely on the id column being filled without gaps. As soon as you delete records from the table, gaps arise that are never filled upon the addition of new records to the table.


Full-Text Search

If you are using SQL to search for a word in a character string, the query is often posed in the following form:

SELECT * FROM tablename WHERE column LIKE '%word%'

This query indeed achieves its goal. The only question is, when? There are few queries that are more time-consuming for MySQL to answer than this. (What is worse is searching for several words, perhaps in several columns.) It is not only that all the records of the table must be read. Additionally, many character string comparisons must be made. A traditional index cannot help you here.

Unfortunately, what is most difficult for the computer is simple for the user, who has become accustomed to using an Internet search engine to input a number of search terms without having to deal with complex search criteria.

If you wish to be able to process such queries efficiently in MySQL, you need a full-text index. This is a particular type of index, one that creates a list of all words that appear in a column of a table. MySQL has supported this type of index since version 3.23.23.

Fundamentals

Creating a Full-Text Index

To provide an existing table with a full-text index, you should execute the following command. You may specify arbitrarily many xxxTEXT and (VAR)CHAR columns.

ALTER TABLE tablename ADD FULLTEXT (columnname1, columnname2)

Of course, new tables can also be generated at once with a full-text index:

CREATE TABLE tablename (id INT NOT NULL AUTO_INCREMENT,
                        column1 VARCHAR(50), column2 VARCHAR(100),
                        PRIMARY KEY (id), FULLTEXT (column1, column2))

Full-Text Search

Fora full-text search, the SQL expression MATCH AGAINST is used:

SELECT * FROM tablename
WHERE MATCH (column1, column2) AGAINST ('word1 word2 word3') > 0.001

The result is that all data records are found in which at least one of the three words word1, word2, word3 is contained. The list of columns in MATCH must correspond exactly to the one with which the index was generated. There is no case distinction in the search criteria in AGAINST. The order of search criteria is also irrelevant. Words with three or fewer letters are generally ignored.

MATCH returns as result a floating-point number whose magnitude reflects the relevance of the result. If no words are found or if the search criteria appear in very many records and are therefore ignored, then MATCH returns 0.

The expression MATCH ... > 0.001 excludes results for which MATCH returns very small values. (This gets rid of statistical outliers and inaccuracies due to floating-point representation.)

A MATCH expression can be used to order results. The following query returns the five best results. The condition must be formulated with HAVING, since WHERE does not accept the alias mtch:

SELECT *,
        MATCH(column1, column2) AGAINST('word1 word2 word3') AS mtch
FROM tablename
HAVING mtch > 0.001
ORDER BY mtch DESC
LIMIT 5

Boolean Search Expressions

By default, all search criteria specified in AGAINST are joined by a logical OR. Since MySQL 4.0.1, you can connect search criteria using a Boolean expression. For this, you must follow the search expression with IN BOOLEAN MODE. Furthermore, the special symbols shown in Table 7-1 can be used in the formulation of the expression:

Table 7-1: Boolean search expressions (IN BOOLEAN MODE)
+word | The word must be contained in the data record. Thus AGAINST('+word1 +word2' IN BOOLEAN MODE) corresponds to a Boolean AND in both search criteria.
-word | The word may not appear in the data record.
˜word | The word should not appear in the record. In contrast to -word, such records are not completely excluded, but are given a lesser value. Thus ˜word is suited to eliminate "noise words," that is, expressions that are more hindrance than help in a search.
<word | Gives the word a lesser standing.
>word | Gives the word a greater standing.
word* | Searches for words that begin with word (e.g., word, words, wordless).
"word1 word2" | This exact word order should be searched. Case-insensitive.
( ) | Parentheses can be used to group expressions. AGAINST('+mysql +(buch book)' IN BOOLEAN MODE) searches for data records that contain mysql and either buch or book.

ABoolean full-text search might look like this:

SELECT * FROM tablename
WHERE MATCH(column1, column2)
      AGAINST('+word1 +word2 -word3' IN BOOLEAN MODE)

A significant drawback to Boolean search is that MATCH returns only 1 or 0, according to whether the search criteria were fulfilled or not. There is no longer an evaluation of relevance.

Remark

Note that in a full-text search, search patterns like LIKE are not allowed. The only exception is the variant word*, wherethe wild card * must be placed at the end of the word.

Full-Text Search over Several Tables

A full-text search can be defined only for a single table. To search for data in linked tables, you will have to equip both tables with a full index. The resulting SQL search queries are, however, complicated to formulate and difficult to optimize for the server. (There is an example in the following section.)

Limitations

  • In order for full-text search to function, there must be at least three records in the table. Experiments with tables that contain only one or two records have ended in failure. In general, MATCH functions better with larger tables than with smaller, for which it occasionally returns results that are difficult to understand.

  • Full-text search is based on entire words. The slightest alteration (plurals, endings, etc.) is considered a different word. This makes searching at times more difficult.

    To find all relevant records, variants have to be given in AGAINST (for example, AGAINST('book books') or AGAINST('book books bookcase bookend') to obtain all records having to do with books. One way out is offered by the wild card *, which, however, can be used only in Boolean mode (AGAINST('book*' IN BOOLEAN MODE).

    A word in the sense of full-text search is considered to be a character string composed of letters, numbers, and the characters ' and _. Fortunately, letters with diacritical marks like üàé are considered letters. Note, however, that expressions like C++ are not considered words and therefore cannot be searched for.

  • Words must be at least four characters in length if they are to be considered. This makes the search for abbreviations like SQL impossible. (This limitation can be removed by a reconfiguration of the server; see below.)

  • Full-text search is case-insensitive (which usually makes sense, but sometimes is a bother).

  • Full-text search is currently available only for MyISAM tables (and not for InnoDB tables).

  • The creation of a full-text index is relatively slow (but nonetheless, much faster than in MySQL 3.23.n). This affects not only the initial creation of an index, but all further changes in the table as well.

  • In MySQL 4.1, full text searches are not possible if a Unicode character set is used.

Full-Text Search for Words of Three Letters

As we have already mentioned, full-text search operates by default only for words of four or more letters. The reason for this is to avoid inflating the index with words like a, and, the, and the like. However, that makes the search for terms like ADO and PHP impossible.

With the configuration parameter ft_min_word_len (mysqld), you can set the minimum length of words for the full-text index. To do this, enter the parameter into the MySQL configuration file (/etc/my.cnf or Windows \my.ini) in the section [mysqld]. For the setting to take effect, you will have to restart the MySQL server and regenerate the full-text index (the easiest way to do this is with REPAIR TABLE tablename QUICK ).

Book Search

For the following examples, the column title of the table titles from the mylibrary database is equipped with a full-text index. Then the best hits of the search for excel or basic are returned.

USE mylibrary
ALTER TABLE titles ADD FULLTEXT (title)
SELECT title, MATCH (title) AGAINST ('excel basic') AS fulltextmatch
FROM titles
HAVING fulltextmatch > 0.001
ORDER BY fulltextmatch DESC

title | fulltextmatch
Visual Basic | 2.7978503848587
Visual Basic Datenbankprogrammierung | 2.7667480431589
VBA-Programmierung mit Excel 7 | 2.3731654830885
Definitive Guide to Excel VBA | 2.3467841568515
Excel 2000 programmieren | 2.3467841568515

In the above query, the condition fulltextmatch > 0.001 was formulated with HAVING, since WHERE does not recognize the alias column name (that is, AS fulltextname). The following variant is a bit more efficient, but is more work to type in:

SELECT title, MATCH(title) AGAINST('excel basic') AS fulltextmatch
FROM titles
WHERE MATCH(title) AGAINST('excel basic') > 0.001
ORDER BY fulltextmatch DESC

Search for Title and Author

A full-text index can be defined only for a single table. If you wish to search for author and title simultaneously (such as excel and kofler), then you must equip all the tables with a full-text index and then in the query specify MATCH expressions for each index. However, the resulting queries are relatively inefficient (especially with large data sets).

USE mylibrary
ALTER TABLE titles ADD FULLTEXT (title)
ALTER TABLE authors ADD FULLTEXT (authName)
SELECT title, authname FROM titles, authors, rel_title_author
WHERE titles.titleID = rel_title_author.titleID
  AND authors.authID = rel_title_author.authID
  AND MATCH (title) AGAINST ('excel kofler')
  AND MATCH (authName) AGAINST ('excel kofler')
title | authname
VBA-Programmierung mit Excel 7 | Kofler Michael
Excel 2000 programmieren | Kofler Michael
Definitive Guide to Excel VBA | Kofler Michael

If titles contained, say, 100,000 titles, and authors as many author names, then the above query would be somewhat slow despite the full-text index. It would be more efficient to store the relevant data in a single table. In the following example titles must be expanded to include an authors column with author names. It is precisely this that was suggested in Chapter 5 as a possible change in database design.

Forum Search

The following examples relate to the table messages in the database bigforum (a variant of the myforum database with about 1300 contributions to a discussion forum). The full-text index was created with the following command:

USE bigforum ALTER TABLE messages ADD FULLTEXT(msgText, subject)

Altogether, there are over 200 contributions that deal with MySQL:

SELECT COUNT(*)FROM messages
WHERE MATCH(msgText, subject) AGAINST('mysql') > 0.001

COUNT(*)

219

MySQL believes that the following entries have the greatest relevance. This result should be taken with a grain of salt: The mechanism of full-text search is not blessed with clairvoyant insight, and the selection from among the many contributions is more or less random, reflecting generally the frequency with which the expression "MySQL" appears:

SELECT subject, msgID,
    MATCH(msgText, subject) AGAINST('mysql') AS mtch
FROM messages
ORDER BY mtch DESC
LIMIT 5
subject | msgID | mtch
help needed with figuring out error message | 332 | 3.222867812652
Re: where is MySQL database? | 359 | 3.1858990010743
Re: Re: Re: MYSQL Zugriff . . . | 357 | 3.0232702237961
Re: An update to the first edition of MySQL? | 775 | 3.0158353640643
Re: Datenbankanbindung | 1262 | 2.8449136579526

The search for the strings mysql and odbc returns only 22 hits:

SELECT COUNT(*)FROM messages
WHERE MATCH(msgText, subject)
AGAINST('+mysql +odbc' IN BOOLEAN MODE)

COUNT(*)

22

To obtain the five best hits, the query has to be changed a bit. As WHERE criterion we have the Boolean search expression '+mysql +odbc', which, however, returns only the values 0 and 1. For sorting, we use instead the expression 'mysql odbc':

SELECT LEFT(subject, 40) AS subj,
    msgID,
    MATCH(msgText, subject) AGAINST('mysql odbc') AS mtch
FROM messages
WHERE MATCH(msgText, subject) AGAINST('+mysql +odbc' IN BOOLEAN MODE)
ORDER BY mtch DESC
LIMIT 5
subj | msgID | mtch
Wie werde ich das mySQL ODBC-Fenster los | 728 | 6.770324495441
Re: Re: Re: Wie werde ich das mySQL ODBC | 842 | 6.453936383473
Re: Re: Wie werde ich das mySQL ODBC-Fen | 734 | 6.093977318182
Datentransfer zwischen Excel und MySQL | 876 | 5.270302575910
Re: Re: Re: MYSQL Zugriff . . . | 357 | 5.251107031037


Chapter 8: InnoDB Tables and Transactions

Overview

By default, mysql uses the myisam table driver. As an alternative, since version 3.23.34 there has been available the considerably more modern InnoDB table driver. InnoDB tables offer, among other things, two functions that MyISAM tables do not (yet) have: transactions and integrity rules (foreign key constraints). The latter of these involves an automatic ensuring of referential integrity in linked tables.

This chapter offers a brief introduction to the world of InnoDB, and with its comparison of MyISAM and InnoDB tables offers a basis for choosing the optimal table type for a particular application. Then there follows an extensive description of transactions and foreign key constraints.

Pointer

Tips on the administration of InnoDB tables, for example, for the management of tablespace files and the InnoDB logging files, can be found in Chapter 10, the chapter on administration. A summary of InnoDB configuration parameters can be found in Chapter 19.


Introduction

Since MySQL 3.23, the default table format has been MyISAM. This format is mature, stable, and fast, and completely satisfactory for many applications. In addition to the MyISAM format, since version 3.23.34, MySQL has supported another, less well known, table format, namely, InnoDB. This is a modern alternative to MyISAM, which is distinguished primarily in that it offers the following supplementary functions:

  • Database operations in InnoDB tables can be executed as transactions. This allows you to execute several logically connected SQL commands as a single entity. If an error occurs during execution, then all of the commands (not only the one during which the error occurred) are nullified. In addition, transactions offer other advantages that improve the security of database applications.

    Transactions can be executed in all four isolation levels of the ANSI-SQL/92 standard (READ UNCOMMITTED, READ COMMITTED, REPEATABLE READ, SERIALIZABLE).

  • In implementing transactions, the InnoDB table driver uses internal row level locking. This means that during a transaction, the entire table does not have to be blocked for access by other users (which would be the case for a MyISAM table during a LOCK TABLE command), but only the data records actually affected. If many users are simultaneously making changes on a large table, row level locking can bring about an enormous advantage in efficiency.

    The InnoDB table driver automatically recognizes deadlocks (that is, the condition in which two processes mutually block each other) and in such a case, terminates one of the two processes automatically.

  • When you define relations between your tables, the InnoDB table driver automatically ensures that the referential integrity of the table is preserved after delete commands. Thus it is impossible, for example, for a record in table A to refer to a no longer existing table B. (In database lingo this function is called a foreign key constraint.)

  • After a crash, InnoDB tables are automatically and very quickly returned to a consistent state (provided that the file system of the computer was not damaged). I have not tested this functionality.

  • Many of the features of the table driver are quite similar to those of Oracle, which should be helpful in porting database applications.

The InnoDB table driver has been an integral component of MySQL since version 3.23.34. The development of the table driver and its commercial support come from the independent company Innobase (see http://www.innodb.com).

It is difficult to say how far InnoDB has met the hopes and expectations of its author. In any event, InnoDB was developed with great diligence and could well quickly become one of the most attractive table formats alongside MyISAM.

Pointer

The use of InnoDB tables is described rather extensively in the MySQL documentation, in the chapter "MySQL Table Types." Further information, including many technical details, can be found at the Innobase web site:

http://www.mysql.com/doc/en/InnoDB.html
http://www.innodb.com/

Limitations and Drawbacks

  • While with the MyISAM table driver, each table is stored in its own file, which grows or shrinks as required, the InnoDB table driver stores all data and indexes in a tablespace, comprising one or more files, that forms a sort of virtual file system. These files cannot later be made smaller. Nor is it possible to stop the MySQL server and then copy a table by simply copying its file. Therefore, in the administration of InnoDB tables, the command mysqldump must be employed more frequently than with MyISAM tables.

  • The storage requirements for InnoDB tables are much greater than those for equivalent MyISAM tables (up to twice as big).

  • ForInnoDB tables one cannot use a full-text index.

  • Indexes for TEXT/BLOB columns are not possible. (This deficiency is supposed to be eliminated soon.)

  • InnoDB supports foreign key constraints, but these can be removed only since MySQL 4.0.13. In older versions, you have to recreate the table and copy all the data into the new table.

  • On account of open transactions, it is relatively difficult for the InnoDB table driver to determine the number of records in a table. Therefore, executing a SELECT COUNT(*) FROM TABLE is much slower than with MyISAM tables. This limitation should be soon eliminated, perhaps before you have this book in your hands.

  • The mysql tables for managing MySQL access privileges cannot be transformed into InnoDB tables. They must remain in MyISAM format.

  • Since InnoDB uses its own locking algorithms in executing transactions, LOCK TABLE should not be used. This command is executed outside the control of the InnoDB table driver by MySQL. It can therefore screw up the locking regime and in the worst case, lead to deadlock (where individual processes wait for each other endlessly).

    Pointer

    Further details on the limitations of InnoDB tables in relation to MyISAM tables can be found at the following address:

    http://www.innodb.com/ibman.html#InnoDB_restrictions

MyISAM or InnoDB?

You can specify individually for each table in your database which table driver is to be used. That is, it is permitted within a single database to install both MyISAM and InnoDB tables. This allows you to choose the optimal table driver for each table, depending on its content and the application that will be accessing it.

MyISAM tables are to be recommended whenever you want to manage tables in the most space- and time-efficient way possible. InnoDB tables, on the other hand, take precedence when your application makes use of transactions, requires greater security, or is to be accessed by many users simultaneously for making changes.

There is no generally valid answer to the question of which table type offers faster response. In principle, since transactions take time and InnoDB tables take up more space on the hard drive, MyISAM should have the advantage. But with InnoDB tables you can avoid the use of LOCK TABLE commands, which offers an advantage to InnoDB, which is better optimized for certain applications.

Furthermore, the speed of an application depends heavily on hardware (particularly the amount of RAM), the settings in the MySQL configuration file, and other factors. Therefore, I can provide here only the following advice: In speed-critical applications, perform your own tests on both types of tables.

Alternatives to InnoDB (BDB, Gemini)

If your main interest is the support of transactions, then instead of InnoDB tables, you could use BDB tables. BDB stands for Berkeley Databases, an open source library for storing data in tables. BDB is not in itself a complete database system, but rather is to be used in combination with MySQL or another program that offers the basic database functions. From a historical point of view, BDB was the first transaction-capable MySQL table type. The first version of MySQL with BDB arrived in 2001.

BDB tables are available only in special compiled versions of MySQL (MySQL Max and MySQL Pro). The next section explains how you can tell whether your version of MySQL supports BDB. With a BDB-capable version of MySQL you can create and use BDB tables more or less as you would InnoDB tables. Note, however, that the BDB table driver manages transactions internally completely differently from the InnoDB table driver. For this reason, there are some significant differences in the behavior of transactions between the two drivers.

Pointer

In contrast to the first edition of this book, this second edition does not go into detail on the subject of BDB tables, since their importance for MySQL has become greatly diminished since the integration of the InnoDB format. The use of BDB tables is described in the MySQL documentation in the chapter "MySQL Table Types." Further information on BDB (independent of MySQL) can be found on the web site of the company SleepyCat:

http://www.mysql.com/doc/en/BDB.html
http://www.sleepycat.com/

In 2001, there appeared a third transactions-capable table type, in addition to BDB and InnoDB, namely, Gemini. The code for the Gemini table driver comes from the Nusphere company (http://www.nusphere.com). Unfortunately, a legal battle between the MySQL company and Nusphere arose over the question of whether Gemini extensions of MySQL were in accord with the MySQL license. In the meantime, the legal issues have been resolved, but with the result that the Gemini table driver vanished into oblivion, and is now to be found neither in the official MySQL code nor on the Nusphere web site.

Determining Which Table Types Your Version of MySQL Supports

To determine which table formats (and other supplementary functions) your version of MySQL supports, execute the following SQL command (for example, in the program mysql):

SHOW VARIABLES LIKE 'have%'
Variable_name | Value
have_bdb | NO
have_crypt | NO
have_innodb | YES
have_isam | YES
have_raid | NO
have_symlink | YES
have_openssl | NO
have_query_cache | YES

We see from this result that InnoDB tables are supported, but not BDB tables. If for have_bdb or have_innodb the value DISABLED is displayed, then the format is supported in principle, but there is a configuration error somewhere, for which reason the function was deactivated when the server was started. Information is generally available in the MySQL logging file (Unix/Linux: /var/lib/mysql/hostname.err, Windows: mysql\data\hostname.err).


First Experiments with InnoDB Tables

As we have already mentioned, all InnoDB tables (regardless of database of origin) are stored in one large file, together with their indexes. This file represents a sort of virtual storage, which is managed by the InnoDB driver itself.

In earlier versions of MySQL with InnoDB support, an explicit configuration for the location and size of the InnoDB file was required. In current versions, however, this is taken care of automatically: When MySQL supports InnoDB, at initial launch the file ibdata1 is created in the data directory with size 10 megabytes. As soon as the InnoDB tables stored in this file require more space, the file is expanded in 8-megabyte quanta. In addition to ibdata1, the logging files ib_logfile0, ib_logfile1, and ib_arch_log_0000000000 are created. All four of these files exist in binary format and should not be changed "by hand."

If you are working with MySQL 3.23n, you will need to add the following lines in /etc/my.cnf or Windows\my.ini and then restart the server:

innodb_data_home_dir = /var/lib/mysql
innodb_data_file_path = ibdata1:20M

Under windows, instead of /var/lib/mysql you specify the MySQL data directory (usually c:/mysql/data). Beginning with MySQL 3.23.50, you can use with innodb_data_file_path the autoextend attribute, which allows the database file to grow automatically:

innodb_data_file_path = ibdata1:10M:autoextend

Note that the InnoDB functions are automatically deactivated when MySQL 3.23n is started unless this minimal configuration has been put in place. Only starting with MySQL 4.0 does InnoDB function without previous configuration. Further information on InnoDB administration can be found in Chapter 10.

Creating InnoDB Tables

From now on, you can generate as many InnoDB tables as you like, which can be placed in existing databases or in new ones. In the following example, a database is created, and within it a new InnoDB table:

CREATE DATABASE innotest
USE innotest
CREATE TABLE table1 (colA INT AUTO_INCREMENT, colB INT,
PRIMARY KEY (colA)) TYPE=InnoDB
INSERT INTO table1 (colB) VALUES (10)

The following commands provide an overview of the files that now exist. The actual data are located in testib. The file innotest/table1.frm contains merely the definition of the table layout (in the same format as for all other MySQL tables):

root# cd /var/lib/mysql
root# ls -l ib* testib
-rw-rw---- 1 mysql mysql 25088 ib_arch_log_0000000000
-rw-rw---- 1 mysql mysql 5242880 ib_logfile0
-rw-rw---- 1 mysql mysql 5242880 ib_logfile1
-rw-rw---- 1 mysql mysql 10485760 testib
root# ls -l innotest/
-rw-rw---- 1 mysql mysql 8578 table1.frm

The above commands were executed under Linux. Under Windows, exactly the same file is created. (The files are transferable among various MySQL installations. You can copy the entire MySQL data directory with all MyISAM and InnoDB tables from Windows to Linux or in the opposite direction.)

Changing a Table Type

You can easily change existing tables from MyISAM to InnoDB (and, of course, from InnoDB to MyISAM). To accomplish this, you use the command ALTER TABLE:

ALTER TABLE tblname TYPE=InnoDB

The alteration fails if the MyISAM table contains a full-text index. This must first be deleted:

ALTER TABLE tblname DROP INDEX indexname

If you wish to change a large number of tables, then you will want to use (under Unix/Linux) the script mysql_convert_table_format. If you do not specify a table name, then all tables of the database will be converted:

root# mysql_convert_table_format [opt] --type=InnoDB dbname [tablename]
Caution

Do not change the table type of the the tables in the mysql database. These tables, which contain management information internal to MySQL (user and access privileges) should remain in MyISAM format.


Transactions

Why Transactions?

Transactions can help in making the operation of a database system more efficient and secure:

  • Transactions ensure that a group of SQL commands that begin with BEGIN and end with COMMIT are executed either as a unit or not at all. Even if the connection with the server is dropped during the transaction, power goes out, or the computer crashes, it cannot happen that only some of the commands are executed.

    Let us look at an example: You wish to transfer 100 dollars from account 123 to account 456. Therefore, you first execute

    UPDATE tablename SET value=value-100 WHERE accountno=123

    and then

    UPDATE tablename SET value=value+100 WHERE accountno=456

  • If you are working without transactions and some misfortune occurs between the two commands, it can happen that 100 dollars was withdrawn from the first account but not deposited to the second. With transactions, that cannot occur: Either both commands are correctly executed, or neither of them.

  • Transactions also ensure that data cannot be simultaneously changed by two users. With MyISAM tables, you can achieve this result with LOCK TABLE. But that has the effect of blocking access to the entire table for all clients. With InnoDB tables, in contrast, only the affected records are blocked.

  • Transactions make for easier programming. A transaction can be aborted at any time, which makes protecting against errors much simpler.

ACID Rules

The database-theoretic answer as to why one should use transactions can be summed up in one word: ACID. We are not talking about controlled substances here, but an acronym: Atomicity, Consistency, Isolation, Durability. Thus ACID encompasses four concepts that are considered in database theory to be the basis of secure simultaneous use of a database by multiple users. MySQL in combination with the InnoDB table driver passes the ACID test, adhering to its rules and regulations, which can be described in somewhat greater detail as follows:

  • Atomicity means that a transaction is indivisible. The database system must ensure that either all the commands of a transaction are correctly executed, or none of them. This holds even for such extreme cases as a crash during execution of the transaction.

  • Consistency requires that at the end of the transaction the database be in a consistent state. If a transaction were to lead to a violation of validity rules, then it must be broken off and undone (ROLLBACK ). Validity rules include foreign key constraints, which will be discussed in greater detail in the following section.

  • Isolation means that several transactions can run independently and simultaneously without interfering with each other. Each transaction sees the entire database in the condition that it found it right up to the end of the transaction (other than those changes made by the transaction). If a transaction inserts, changes, or deletes records, a transaction running in parallel does not see this.

    Complete fulfillment of the isolation requirement can be achieved only at great cost, and in practice, this means cost in speed. For this reason, the ANSI-92/SQL standard provides for four degrees of isolation, so that database programmers can decide on the appropriate level of security and the tradeoff against speed. The default isolation level depends on the database system, and for the InnoDB table driver it is REPEATABLE READ. We will have more to say later on isolation degree.

  • Durability requires that a transaction itself be preserved even if right after the end of the transaction there is a crash or other comparable problem. (The InnoDB table driver therefore writes changes first in a logging file. If a crash occurs, that is, before the changes can actually be made to the database, the changes can be reconstructed after restart of the MySQL server and transferred to the database.

    Many database systems make compromises in their durability requirements to achieve improvements in speed. In the case of the InnoDB table driver, most significant in this regard is the option innodb_flush_log_at_trx_commit, which controls when transactions can be stored in the logging file.

Controlling Transactions

By default, MySQL is in auto commit mode. This means that every SQL command is executed in a single, small transaction. This has no effect on the use of transaction-capable tables.

There are two ways of executing several SQL commands together as a transaction:

  • You can start an ad hoc transaction with BEGIN and terminate it with COMMIT or ROLLBACK. BEGIN deactivates the auto commit mode for this one transaction. If you subsequently wish to begin a transaction, then you must begin with another BEGIN.

    Since MySQL 4.0.11 you can introduce an ad hoc transaction with START TRANSACTION (instead of with BEGIN). START TRANSACTION conforms to the ANSI standard, but otherwise, is no different from BEGIN.

  • You can turn off auto commit mode. Then all the commands are considered together as a transaction until confirmed by COMMIT or aborted by ROLLBACK.

BEGIN, COMMIT, and ROLLBACK

BEGIN or (since MySQL 4.0.11) START TRANSACTION introduces a transaction, COMMIT ends it and stores all changes, and ROLLBACK aborts the transaction and stores no changes.

In InnoDB tables one may not have nested transactions. If you begin a new transaction with BEGIN while a previous transaction is still open, the open transaction is closed with COMMIT.

Transactions are managed by the client. If the connection is lost during a transaction, then all uncommitted changes are aborted (as with ROLLBACK ).

Savepoints

Since MySQL 4.0.14, InnoDB has supported savepoints. This allows points within a transaction to be marked using SAVEPOINT name. Then ROLLBACK TO SAVEPOINT name ends the transaction, accepting all changes up to the savepoint and aborting all those that follow. Savepoints can be used only within a transaction, and at the end of the transaction, all savepoints are deleted.

Automatic Transaction Termination

Transactions are also concluded automatically by the following commands (as with COMMIT):

ALTER TABLE, CREATE INDEX, CREATE TABLE, DROP DATABASE,
DROP TABLE, LOCK TABLES, RENAME TABLE, SET AUTOCOMMIT=1,
TRUNCATE, UNLOCK TABLES

On the other hand, transactions are not ended when an ordinary error occurs in the execution of an SQL command.

Setting Auto Commit Mode

The auto commit mode is deactivated with the SQL command SET AUTOCOMMIT = 0. Commands that relate to transaction-capable tables will now automatically be considered a transaction until they are ended with COMMIT or ROLLBACK. Then a new transaction begins automatically. (That is, you do not need to execute a BEGIN. Just this convenience is often reason enough to work with SET AUTOCOMMIT = 0.)

An important consequence of AUTOCOMMIT = 0 is that if the connection is lost, whether or not intentionally, all SQL commands that have not been confirmed with a COMMIT are aborted.

Note also that AUTOCOMMIT=0 can lead to very long transactions if you do not regularly execute COMMIT or ROLLBACK. The current state of auto commit can be determined with SELECT @@autocommit.

Controlling Transactions in Client Programming

Most libraries and APIs for MySQL programming (such as JDBC, ODBC, ADO, ADO.NET) provide special functions or methods for beginning and ending transactions. However, the InnoDB documentation suggests that you not rely on such functions but that you explicitly execute the commands SET AUTOCOMMIT and BEGIN/COMMIT/ROLLBACK. The reason is that with many APIs the use of transaction functions in combination with MySQL has not been sufficiently tested and does not always work as advertised.

Trying Out Transactions

To try out transactions, you must establish two connections to the database. The simplest way to do this is to execute the monitor mysql in two windows. Then execute commands first in one window and then in the other. The following examples assume that there is a (still empty) table table1 in the database innotest.(In practice, of course, you will have several records, but for the purposes of this first demonstration, even a single record suffices.)

Remark

This example shows the effect that transactions can have when data in a table are changed nearly simultaneously by two users. The example is very simple, and the simultaneity takes place in slow motion. Nonetheless, it should make you aware that in real applications it can certainly happen that database commands from two users or programs can be woven together as in the following example. If such an effect on the end result is possible, then you must exclude the possibility by the use of transactions or (if you wish to stick with MyISAM tables) by executing LOCK TABLE.

At time point 0, connection A begins a transaction (BEGIN) and changes, in the confines of this transaction, a record of the table table1.

At time point 1 (as indicated in the right-hand column of Table 8-1) the contents of table1 look different to the two different connections. For connection A, the record with colA=1 already has colB=11.Since this transaction has not actually been executed, connection B sees the original value colB=10 for the same record.

Table 8-1: Coordinating transactions
Connection A | Connection B | Time point
USE innotest | USE | innotest
INSERT INTO table1 VALUES (1, 10) |  |
SELECT * FROM table1 colA colB 1 10 |  |
BEGIN |  | 0
UPDATE table1 SET colB=11 WHERE colA=1 |  |
SELECT * FROM table1 colA colB 1 11 | SELECT * FROM table1 colA colB 1 | 1 10
 | BEGIN UPDATE table1 SET colB=colB+3 WHERE colA=1 | 2
COMMIT |  | 3
SELECT * FROM table1 colA colB 1 11 | SELECT * FROM table1 colA colB 1 | 4 14
 | ROLLBACK |
SELECT * FROM table1 colA colB 1 11 | SELECT * FROM table1 colA colB 1 | 5 11

At time point 2, B begins a transaction, in which colB of the data record with colA=1 is to be increased by 3. The InnoDB table driver recognizes that it cannot execute this command at this time, and it blocks B, which then waits until A completes its transaction.

At time point 3, A completes its transaction with COMMIT, whereby colB attains the definitive value 11. And now B's UPDATE command can be completed.

At time point 4, A sees colB=11. For B things look as though colB already had the value 14.

Now B cancels its transaction. Then at time point 5 both A and B see the value actually stored, namely colB=11.

Remark

Of course, you can try out this example with BDB tables. The result of the commands will be the same (if you maintain the sequence of execution). However, it can happen that blocking comes at other places.

Transactions and Locking

The InnoDB table driver generally takes care of all necessary locking operations on its own, as soon as you execute your transactions. However, there are cases where the default behavior of InnoDB is not optimal. In such cases, InnoDB offers several possibilities of having its locking behavior altered.

SELECT ... LOCK IN SHARE MODE

A peculiarity of the InnoDB table driver is that SELECT commands are immediately executed even on blocked records. The results returned do not consider the possibility of open transactions of other clients (see connection B at time point 1 in the example above), and thus return potentially outmoded data.

If you execute the SELECT with the key word LOCK IN SHARE MODE appended, then when the command is executed, it is held pending until all transactions already begun have been terminated (of course, only to the extent that these transactions affect result records of the SELECT command). Thus if in the above example B were to issue the command

SELECT * FROM table1 LOCK IN SHARE MODE

at time 1, then a result would be displayed only when the transaction begun by A had been completed (time point 3).

If SELECT ... LOCK IN SHARE MODE is executed within a transaction, then additionally, all result records for all other clients will be locked until the end of the transaction. Such a lock is called a shared lock, whence the key word SHARE. With a shared lock you are assured that the records read during your transactions are not being changed or deleted by other clients.

With a shared lock, locked records can continue to be read by all clients, even if other clients are also using SELECT LOCK IN SHARE MODE. Any attempt by a client to change such records leads to the client being blocked until your transaction is completed.

SELECT ... FOR UPDATE

The key words FOR UPDATE also represent an extension of the normal SELECT. With this, all result records are provided an exclusive lock.

With an exclusive lock, locked records cannot be changed by other clients. They can continue to be read by all clients with a normal SELECT command, but not with SELECT ... LOCK IN SHARE MODE. The difference between a shared lock and an exclusive lock therefore relates only to whether other clients can execute SELECT ... LOCK IN SHARE MODE.

INSERT, UPDATE, DELETE

All three of these commands have the effect that changed records are locked by an exclusive lock until the end of the transaction.

If links between tables must be checked (foreign key constraints) during the execution of an INSERT, UPDATE, or DELETE command, then the affected records of the linked tables will be locked by an exclusive lock.

Gap and Next Key Locks

InnoDB uses by default gap and next key locks when open conditions are used with SELECT ... LOCK IN SHARE MODE, SELECT ... FOR UPDATE, UPDATE, or DELETE (e.g., WHERE id>100 or WHERE id BETWEEN 100 AND 200). The result is that not only the records currently affected by the condition are blocked, but also records that do not yet even exist that might be inserted by another transaction.

Thus if you execute SELECT ... WHERE id>100 FOR UPDATE in a transaction, then no other user can input new records with id>100 for the duration of the transaction.

Pointer

Which commands have next key locks placed on them is also affected by the isolation degree. See the following section.
Deadlocks

The InnoDB table driver automatically recognizes deadlock situations (that is, situations in which two or more processes block one another, each waiting for the other to finish, ad infinitum). In such situations an error occurs in the process that triggered the deadlock; SQL commands that are still open are aborted with a ROLLBACK.

Caution

In no case should you execute a LOCK TABLE for an InnoDB table. This command is executed at the MySQL level, without the InnoDB table driver hearing about it. The consequence can be a deadlock of which InnoDB never becomes aware.

Deadlocks that InnoDB doesn't know about can also arise when SQL commands affect both InnoDB tables and other tables as well. In order for the clients not to have to wait forever in such cases, the configuration setting innodb_lock_wait_timeout=n determines the maximum wait time (by default, 50 seconds).

Isolation Degree for Transactions

Before the start of a transaction, its isolation level can be defined:

SET [SESSION|GLOBAL] TRANSACTION ISOLATION LEVEL
  READ UNCOMMITTED | READ COMMITTED |
  REPEATABLE READ | SERIALIZABLE

SET can be executed in three different ways:

  • SET without SESSION or GLOBAL: In this case, the selected setting holds only for the next transaction. Note, please, that this behavior is different from that of the usual SET syntax, where SET works without the specification of what is to be set, as in SET SESSION.

  • SET SESSION: Here, the setting holds for the current connection until its termination or until another isolation degree is specified.

  • SET GLOBAL: Here, the setting holds for all new connections to MySQL (but not for the current connection).

The isolation degree influences the manner in which commands are executed within a transaction. In the following list, the isolation degrees are ordered according to increasing exclusivity. This means that you achieve the greatest access speed with READ UNCOMMITTED (no mutual blocking), while with SERIALIZABLE, you obtain the greatest security against simultaneous changes of data by clients. We now describe the four isolation degrees:

  • READ UNCOMMITTED: SELECT reads the current data and considers all changes arising from other running transactions. SELECT is therefore not isolated from other transactions. (The expression read uncommitted means that data of other transactions that are not yet completed with COMMIT can flow into the SELECT result.) If you execute the example from Table 8-1 in READ UNCOMMITTED mode, then at time point 1, transaction B sees colA=1 and colB=11.

    Please note that while READ UNCOMMITTED does not ensure isolation for SELECT, it does so for UPDATE. Even in READ UNCOMMITTED mode, the UPDATE command of transaction B is blocked at time point 2 until transaction A is finished. UPDATE is therefore correctly executed in READ UNCOMMITTED mode.

  • READ COMMITTED: SELECT takes into account changes by other transactions that have been confirmed with COMMIT. This means that one and the same SELECT command within a transaction can have different results if another transaction is closed in the intervening time. In comparison with READ UNCOMMITTED, the isolation for SELECT is somewhat better, but not perfect.

  • REPEATABLE READ: SELECT takes into account no changes by other transactions, regardless of whether these transactions have been confirmed with COMMIT. Thus the ACID isolation requirement for SELECT is satisfied for this isolation degree. One and the same SELECT command always returns one and the same result (as long, of course, as the transaction does not itself change data).

  • SERIALIZABLE: This mode functions much like REPEATABLE READ. The only difference is that ordinary SELECT commands are automatically executed as SELECT ... LOCK IN SHARE MODE, and therefore supply all records with a shared lock.

By default, InnoDB uses the isolation level REPEATABLE READ. In most cases, this mode offers sufficient isolation for a variety of transactions without influencing transaction speed unduly. InnoDB is optimized for REPEATABLE READ mode and executes the required locks especially efficiently.

One can change the default setting in the MySQL configuration file with the option transaction-isolation. In setting the option, the names of the isolation levels consisting of more than one word must be separated by hyphens (thus, for example, transaction-level = read-committed).

You can determine the isolation level of a connection or of the server with the following command:

SELECT @@tx_isolation, @@global.tx_isolation

@@tx_isolation | @@global.tx_isolation
READ-UNCOMMITTED | REPEATABLE-READ

Isolation Level and Next Key Locks

As we have already mentioned, InnoDB supports gap and next key locks. This means that in open conditions (e.g., WHERE id>100), not only currently existing records are blocked, but also data that do not yet exist that would satisfy the condition.

InnoDB uses gap and next key locks only when open conditions appear in commands. Moreover, the locking is also dependent on the command being executed and on the isolation level. The following conditions hold in MySQL 4.0:

  • READ COMMITTED and READ UNCOMMITTED: Gap and next key locks are used only for commands of the form INSERT INTO ... SELECT ... FROM s ... .

  • REPEATABLE READ and SERIALIZABLE: Gap and next key locks are used for the following commands: INSERT INTO ... SELECT ... FROM s ... , SELECT ... LOCK IN SHARE MODE, SELECT ... FOR UPDATE, UPDATE, DELETE.

    Starting with MySQL 4.1, it is planned that no locking will take place with the command INSERT INTO ... SELECT ... FROM s ... . Instead, the command will be considered a consistent read command. The table driver then ensures (depending on isolation level) that a repeated execution of the command leads to the same result.

Error Protection

When you use transactions, you increase the odds of receiving certain error messages from the server. You should therefore always check in your code, even with apparently noncritical commands (such as SELECT), whether errors have occurred during execution:

  • While a transaction is open, another client cannot execute any operation that affects the records of the open transaction.

    Example: Client A executes BEGIN and then INSERT INTO table VALUES (1, 2).

    Client B attempts to execute SELECT * FROM table. This command is blocked until client A terminates its transaction with COMMIT or ROLLBACK.

    Many clients, such as mysql, will wait forever for an open transaction to be completed (there is no timeout). But other clients can experience a timeout or other error. In sum, close your transactions as soon as possible.

  • Every transaction can be broken off due to a deadlock situation. (The InnoDB table driver automatically recognizes deadlock situations, that is, the situation in which two transactions mutually block each other. To avoid a permanent blocking of the server, one client's transaction will be terminated.)


Integrity Rules (Foreign Key Constraints)

InnoDB is the first MySQL table driver that is capable of ensuring the referential integrity of linked tables through integrity rules (foreign key constraints). For example, in the mylibrary database you can thereby make it impossible to delete a publisher from the publishers table as long as there is at least one book title in the titles table that was published by that publisher.

The following commands show how a foreign key constraint can be employed after the fact using ADD FOREIGN KEY. (The key here is the second command. The first command merely creates an ordinary index for the column titles.publID, which is aprecondition for the following foreign key constraint.)

ALTER TABLE titles ADD INDEX (publID) ALTER TABLE titles ADD FOREIGN KEY (publID)
REFERENCES publishers (publID)

Of course, foreign key constraints can also be given when a table is created, using CREATE TABLE,asin the following:

CREATE TABLE titles (
titleID int(11) NOT NULL AUTO_INCREMENT
...
PRIMARY KEY (titleID),
KEY (publID),
FOREIGN KEY (publID) REFERENCES publishers (publID)
) TYPE=InnoDB

The creation of foreign key constraints has consequences for both tables:

  • In titles you cannot insert a title with a publID number that does not exist in the publishers table. (Nor can you change an existing value of publID for a title in the titles table if there is no corresponding publishers record.)

  • You may not delete a publisher from publishers that is referred to in the titles table. (This restriction holds as well for UPDATE commands.)

These conditions have consequences for the order of operations: If you wish to store a new title for a new publisher, you must first enter the publisher, and only then the title. (You have already done this, because you needed the ID number of the publisher.) If you later wish to delete the publisher and the title, you must first delete the title and then the publisher. (Deleting the publisher is possible only if there are no other titles for that publisher.)

Syntax

The general syntax for defining a foreign key constraint for a foreign key field table1.column1 looks as follows:

FOREIGN KEY [name] (column1) REFERENCES table2 (column2)
  [ON DELETE {CASCADE | SET NULL | NO ACTION | RESTRICT}]
  [ON UPDATE {CASCADE | SET NULL | NO ACTION | RESTRICT}]

The foreign key constraint can be named with name. The foreign key is table1.column1 for which the constraint is defined. The field of the second table to which the constraint refers is table2.column2. (In many cases, column2 is the primary key of table2. This is not requisite, but column2 must be equipped with an index.)

One may also have foreign key constraints in which table1 and table2 are the same table. This makes sense when a table possesses references to itself. This is the case, for example, with the categories table in mylibrary, where parentCatID refers to CatID, thus creating a hierarchical relationship among the categories.

Remark

The MySQL documentation provides the syntax CREATE TABLE table1 (column1 datatype REFERENCES table2(column2) ... ) for CREATE TABLE . This syntax is currently ignored, regardless of which table driver you are using.

Actions When Integrity Is Damaged

The optional clause ON DELETE determines how the table driver should behave if a record that is referred to by table1 is deleted from table2. There are four possibilities:

  • RESTRICT is the default behavior. The DELETE command causes an error, and the record is not deleted. (An error does not necessarily mean the termination of a running transaction. The command is simply not executed. The transaction must, as usual, be terminated with a COMMIT or ROLLBACK.)

  • SET NULL has the effect that the record from table2 is allowed to be deleted. In table1, all records that refer to the deleted record will have column1 set to NULL. This rule assumes that NULL is a permitted value for table1.column1.

    For the titles/publisher example, this means that if you remove publisher x from publishers, then all records in titles that were published by this publisher will have their publID set to NULL.

  • CASCADE has the effect that the record from table2 is allowed to be deleted. At the same time, however, all records from table1 that refer to that record will be deleted.

    For the titles/publishers example, this means that if you delete publisher x from the publishers table, then all records in titles that were published by publisher x will be deleted from titles.

  • NO ACTION has the effect that the loss of referential integrity is tolerated. This action seldom makes sense, since it is easier simply not to use a foreign key constraint.

    These four actions can also be specified analogously for ON UPDATE (where by default, RESTRICT again holds). The UPDATE rules come into effect if in table2, the key field of an existing record is altered. In such a case, the effect of RESTRICT, SET NULL, and NO ACTION is the same as with ON DELETE.

On the other hand, the effect of CASCADE is a bit different: A change in the key field in table2 is now also carried out in the foreign key field in table1. For the titles/publishers example, this means that if you change the publID field for publisher x in publishers, then the publID field will be updated for all affected records in titles.

Remark

Integrity rules do not prevent you from deleting an entire table. For example, you can execute DROP TABLE publishers,evenifthismeansdamagingthe referential integrity.

Conditions for Setting Up Integrity Rules

Foreign key constraints can be used only if a set of preconditions is satisfied. If such conditions have not been satisfied, then the result is an error message, generally Error 1005: Can't create table xxx (errno: 150), and the constraint is not stored. (The cause of the error can be quite trivial, such as a typo in the name of a column.)

  • Each of table1.column1 and table2.column2 must be equipped with at least an ordinary index. This index is not created by FOREIGN KEY, and must therefore be explicitly provided for in CREATE TABLE or after the fact with ALTER TABLE.

    table2.column2 is often the primary key field of table2, but that is not necessary.

    If you use keys across several fields (INDEX(columnA, columnB)), then the key field from the foreign key constraint must appear first. Otherwise, an additional separate index for the field must be created.

  • The data types of table1.column1 and table2.column2 must agree to the extent that a direct comparison is possible without transformation of data types. Most efficient is if both fields are declared with INT or BIGINT. Both columns must be of the same sign type (SIGNED or UNSIGNED).

  • If the optional rule ON DELETE/UPDATE SET NULL is defined, then the value NULL must be permitted in table1.column1.

  • The foreign key constraint must be satisfied from the start: If the tables are already filled with data, it can happen that individual records do not conform to the integrity rules. In this case, an ALTER TABLE command results in error 1216 (A foreign key constraint fails). The records must be corrected before the constraint can be set up.

    Tip

    If an error occurs in setting up a foreign key constraint, you can obtain more precise information, since MySQL 4.0.13, about the cause of the error with SHOW INNODB STATUS.

Finding Unsatisfied Integrity Rules

The obvious question upon receipt of error 1216 is, "How can the error-ridden records be found?" The solution to this conundrum lies in Chapter 7, where it was demonstrated on the tltles/publishers connection how a simple SELECT command determines all records in the titles table for which titles.publID contains a value for which there is no corresponding value in publishers.publID:

SELECT titleID, titles.publID, publishers.publID
FROM titles LEFT JOIN publishers
ON titles.publID = publishers.publID
WHERE ISNULL(publishers.publID) AND NOT ISNULL(titles.publID)
titleID | publID | publishers.publID
66 | 99 | NULL

The title with titleID=66 thus refers to a publisher with publID=99 in the publishers table, but there is no such ID number in that table. Now you must either insert the missing publisher or delete the title with ID equal to 66. (In the titles table you could also change erroneous publIDs to NULL, though in practice, that is often not allowed in linked tables.)

Since MySQL 4.1 you can formulate the same query with a subSELECT. The logic of the query is then easier to understand:

SELECT titleID, publID FROM titles
WHERE publID NOT IN (SELECT publID FROM publishers)

Deleting Foreign Key Constraints

In MySQL 4.0 it was possible already to define foreign key constraints for InnoDB tables, but through version 4.0.12 it was impossible to delete them. The only solution was to recreate the table. The necessary commands (here for the titles table) are shown below. The syntax for CREATE TABLE can be retrieved with SHOW CREATE TABLE oldtitles:

RENAME TABLE titles TO oldtitles CREATE TABLE titles (
titleID INT NOT NULL AUTO_INCREMENT
title VARCHAR(100) NOT NULL,
... comment VARCHAR(255),
PRIMARY KEY (titleID)
) TYPE=InnoDB INSERT INTO titles SELECT * FROM oldtitles DROP TABLE oldtitles

Since MySQL 4.0.13 it has been easier to delete foreign key constraints. You can use the following command:

ALTER TABLE tablename DROP FOREIGN KEY foreign_key_id

You can determine the foreign_key_id of the index to be deleted with SHOW CREATE TABLE.

Deactivating Integrity Checks

With SET foreign_key_checks=0 you can turn off the automatic checking of integrity rules. This can make sense, for example, to speed up the reading in of large backup tables.

Integrity Rules for the mylibrary Database

In the accompanying files for this book you will find a variant of the mylibrary database, namely, mylibraryinno. As revealed by its name, all the tables of this database are in InnoDB format. Moreover, the following foreign key constraints have been defined. (Before this conversion could be done, a large number of new indexes had to be created and a few invalid records corrected.)

Foreign key | Referential key
titles.publID | publishers.publID
titles.langID | languages.langID
titles.catID | categories.catID
categories.parentCatID | categories.catID
rel_title_author.titleID | titles.titleID
rel_title_author.authID | authors.authID


Chapter 9: Security

Overview

It is a fact of life, or at least of human social organization, that not all information is intended to be made available to all individuals. Thus with MySQL, a database is generally set up in such a way that not everyone can see all of the data (let alone change or delete it). In order to protect data from prying eyes (or unauthorized tampering), MySQL provides a dual access system. The first level determines whether the user has the right to communicate with MySQL at all. The second level determines what actions (such as SELECT, INSERT, DROP) are permitted for which databases, tables, or columns.

This chapter describes rather extensively the access system of MySQL, both its internal management and the tools that can assist you in changing access privileges.


Introduction

Normally, it is undesirable to allow everyone to execute all database operations. To be sure, there must be one, or perhaps even several, administrators who have wide-ranging powers. However, it would be a great security risk to allow all users to act as they please in altering records or even deleting entire databases.

There can be many different degrees of access rights. In a company with an employee database, for example, it may make sense to allow all employees to read part of the database (for example, to find someone's telephone number), while other parts remain invisible (such as personnel records).

MySQL offers a finely meshed system for setting such access privileges. In the MySQL documentation this system is called the access privilege system, while the individual lists of the system are called access control lists (ACLs). The management of these lists is carried out internally in the tables of the mysql database, which will be described in detail.

Setting the Access Privileges

There are several ways to set access privileges:

  • The simplest and most convenient way is to use an administration program with a graphical user interface. Some of these programs were described in Chapter 4. Note, however, that even the most convenient user interface is of little help if you do not understand the concepts of MySQL access.

  • You can change mysql directly with INSERT and UPDATE commands.

  • You can use the SQL commands GRANT and REVOKE, which offer greater convenience.

  • You can use the Perl script mysql_setpermission.pl. This script is even easier to use than GRANT and REVOKE, though it assumes, of course, that you have a Perl installation up and running.

User Name, Password, and Host Name

MySQL's access system has two phases. In the first phase, the question is whether the user is permitted to make a connection to MySQL at all. (Such does not imply the right to read or alter any databases, but nevertheless, such access provides the capability to pass SQL commands to MySQL. Then MySQL decides, based on its security settings, whether this command should actually be executed.)

While you are probably accustomed to providing a user name and password to enter the operating system of a multiuser computing system, MySQL evaluates a third piece of information: the name of the computer (host name) from which you are accessing MySQL. Since MySQL's entire security system is based on this informational triple, it seems not inappropriate to take a few moments now to explain what is going on:

User Name: The user name is the name under which you announce your presence to MySQL. The management of MySQL user names has nothing to do with login names managed by the operating system. It is, of course, possible, and indeed, it is often advisable, to use the same name for both purposes, but management of each of these names is independent of the other. There is no mechanism to synchronize these operations (such that, for example, a new MySQL user name is generated when the operating system generates a new user name).

The user name can be up to sixteen characters in length, and it is case-sensitive. In principle, such names do not have to be composed exclusively of ASCII characters. But since different operating systems handle special characters differently, such characters often lead to problems and therefore should be avoided.

Password: What holds for user names holds for passwords as well. There is no relation between the MySQL password and that for the operating system, even if they are identical. (Within MySQL, passwords are stored in 16-bit encrypted form, which allows passwords to be checked, but does not allow reconstruction of a password. Even if an attacker should gain access to the user table of MySQL, there is no way for the villain to determine the passwords themselves. Since MySQL 4.1 a stronger form of encryption has been used, with a code length of 45 characters.)

Passwords, like user names, are case-sensitive, and they can be of arbitrary length. Special characters outside of the ASCII character set are possible, but are not recommended.

Caution

For security reasons do not use the same password for MySQL as for the operating system!

MySQL passwords must often be supplied in plain text in script files, configuration files, and programs. Thus they are in danger of being intercepted. If an attacker who acquires such a password were to gain access not only to your database, but also discovers that the same password allows entry into the operating system, the villain will be very happy indeed.

This warning is directed particularly at Internet service providers that allow their clients direct access to the computer (FTP, Telnet, SSH) in addition to access to a MySQL database. Be sure to give different passwords for these two different forms of access!

Host Name: In establishing a connection, you must normally specify the computer on which the MySQL server is running. This computer name is generally referred to as hostname. The host name can be given as an IP number (e.g., 192.168.23.45). You can omit specification of the host name only if the MySQL server is running on the same computer as your client program.

In access checking, MySQL evaluates the information as to where the access request is coming from, that is, from what host name:

  • The MySQL server must be capable of resolving the computer name. In establishing a connection, the server first receives from the client its IP number. Then the server attempts to find the associated computer name (by contacting a name server). If that is unsuccessful, then the computer uses the IP number instead of the name for determining whether access should be allowed.

  • Even if the name resolution succeeds (so that a local connection is established), there can be complications. If you are working under Unix/Linux, there are two variants of how communication proceeds between client and server:

    • Usually, for reasons of efficiency, a local connection between client and server takes place over a socket file.In this case, localhost (the usual name of the local computer in a network) is the host name. ◦

    • Only when the connection takes place over TCP/IP is the actual computer name used. This holds especially for Java programs, whose database driver does not support socket files.

    Pointer

    Extensive background information on these special cases and various other problems with host names and generally in establishing connections can be found later in this chapter.

Default Values: If no other parameters are given in establishing a MySQL connection, then under Unix/Linux the current login name is given as user name, while under Windows it is the character string ODBC. As password an empty character string is passed. The host name is localhost.

You can easily test this. Launch the program mysql and execute the command status. The current user name is displayed in the line current user. The following commands were executed under Linux:

kofler:~ > mysql -p
Enter password: xxx
Welcome to the MySQL monitor.
mysql> status
...
Current user:
Connection:
kofler@localhost
localhost via UNIX socket
...

If in establishing a connection you must provide a user name other than the login name, a host name, and a password, then the start of mysql looks something like this:

kofler:~ > mysql -u surveyadmin -h uranus.sol -p
Enter password: xxx
Welcome to the MySQL monitor.
mysql> status
...
Current user:
Connection:
surveyadmin@saturn.sol
uranus.sol via TCP/IP
...

Here we have a TCP/IP connection between the client (computer saturn.sol) and the server (computer name uranus.sol).

Remark

MySQL recognizes the term anonymous user. This expression comes into play when any user name is permitted in establishing a connection to MySQL. (In the Windows default configuration this is precisely what is allowed.) In this case, status shows as current user the user name used when the connection was established, but internally, an empty character string is used as user name. This is important primarily for the evaluation of additional access privileges. Further background information is contained later in this chapter, where the inner workings of the access system are discussed.

When a PHP script establishes the connection to the database and no other user name is supplied within mysql_connect() (or mysql_pconnect()), then what is used as user name is the name of the account under which the PHP interpreter is executed. (As a rule, this is the same account under which Apache is also executed. Thus for security reasons, most Unix/Linux systems use an account that has few privileges, such as apache or wwwrun.)

Default Security Settings

If you have newly installed MySQL on a test system, then you are starting out with an immensely insecure default setting, one that is not password-protected. It is interesting to note that this default setting depends on the operating system. Under Windows, the setting is even more insecure than under Unix and Linux. Please note that the default privileges have changed repeatedly over time. The specifications here relate to version 4.0.12.

Unix/Linux: The user root on the local system (host name localhost or computername) has unlimited rights. Furthermore, all users on the local system are permitted to access MySQL without a password; in contrast to root, these users have no rights whatsoever (they can't even execute a SELECT command). Access from an external computer is by default not allowed.

Windows: The user root has, as under Unix/Linux, unlimited rights. However, the security dilemma under Windows is more aggravating, since root is allowed access not only from the local system, but from an external computer in the network.

Moreover, all users of the local system can obtain access under an arbitrary name and have extensive access privileges; for example, they can read all databases, as well as alter them and completely delete them. Finally, anyone can log in (any user name, any host name), although such users, at least in this case, have no rights, and thus can do little after establishing a connection.

Caution

To make things perfectly clear: According to the MySQL default installation, root is not protected by a password. Under Unix/Linux anyone with access to the local system can connect to MySQL as root and do whatever he or she wants with your databases. Under windows it is not even necessary to use the local computer: A connection as root is possible over a network. In the next section, on first aid, you will learn how with a little effort you can replace this default setting with something more restrictive.
Remark

For those with little experience with Unix/Linux, root under Unix/Linux plays somewhat the same role as Administrator under Windows, that is, a superuser with almost unrestricted privileges. However, be advised that user names under MySQL havenoconnectionwiththeloginnames for the operating system. Thus root has so many privileges under MySQL because the MySQL default setting was chosen, not because under Unix/Linux root has such privileges. It would have been possible to choose a name other than root for the MySQL user with administrative access rights.

You may perhaps be asking yourself where the default security settings come from. These settings are stored in the mysql database, which under Windows is created by the MySQL setup program setup.exe. Under Unix/Linux, the script mysql_install_db is executed during RPM installation, which creates and sets up the mysql database.

Under Unix/Linux one can execute mysql_install_db manually, for example, to re-create the mysql database. To do this, the MySQL server must first be stopped. The script must be executed under the same account that is used for executing the MySQL server (usually mysql). You can execute mysql_install_db also as root, in which case you must be the owner of the mysql database directory and change the following files contained therein:

root# mysql_install_db
root# chown mysql -R /var/lib/mysql/mysql

Test Databases

Regardless of the operating system under which MySQL is installed, any user who connects to MySQL can create test databases. The only condition is that the name of the database must begin with the letters test (with older MySQL installations under Unix/Linux it must begin with test_). Please note that not only can every MySQL user create such a database, in addition, every user can read, alter, and delete all data in such a database. In fact, anyone can delete the entire database. In short, the data in test databases are completely unprotected.

Pointer

Tables 9-3, 9-4, and 9-6 depict this default setting. To understand these figures it is necessary that you understand the mysql table functions user and db, which are described in this chapter.


First Aid

Perhaps you are not quite ready to immerse yourself in the depths of SQL access management. In that case, this section provides a bit of first aid advice.

The instructions assume that you communicate with the MySQL server with the commands mysql and mysqladmin. You can, of course, execute equivalent commands with a graphical user interface.

Protecting the MySQL Installation Under Unix/Linux

The following commands assume that MySQL has been freshly installed and that no changes have been made to the default access privileges.

root Password for Local Access

With the following two commands you can secure root access with a password xxx. (Instead of xxx,you should, of course, provide the password of your choice.)

root# mysqladmin -u root -h computername password xxx
root# mysqladmin -u root -h localhost password xxx

Instead of computername you should give the name of the local computer (which you can determine with the command hostname). As we have mentioned, the name localhost is valid when the local connection is effected over a socket file (which is the usual state of affairs). On the other hand, computername holds when the local connection is via TCP/IP (in particular, with Java programs).

No Local Access Without Password: Please note that in accordance with the MySQL default settings, a MySQL connection can be made from the local computer with any user name other than root. In contrast to the root connection, such connections are, to be sure, provided with no privileges, but nonetheless they represent a security risk. (For example, such users have unrestricted access to test databases.)

However, here simple password security with mysqladmin is impossible, since there is no user name to which the password can be assigned. However, you can delete the two relevant entries in the user table of the mysql database. (This has the further advantage that in the evaluation of access privileges an invalid entry will not accidentally be introduced.) The necessary commands look like this:

root# mysql -p
Enter password: xxx
Welcome to the MySQL monitor. ...
mysql> USE mysql;
Database changed
mysql> DELETE FROM user WHERE user='';
Query OK, 2 rows affected (0.00 sec)
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.01 sec)
mysql> exit

Please note that in the future you will have to provide the password that was defined with mysqladmin at the beginning of this section to execute mysql and mysqladmin (option -p). FLUSH PRIVILEGES is necessary so that the changes carried out directly in the mysql database become effective. (MySQL keeps a copy of the mysql database in RAM, for speed optimization, which is updated via FLUSH PRIVILEGES. Instead of executing FLUSH PRIVILEGES, you can exit mysql and instead execute mysqladmin reload.)

Result: After execution of the above commands, only root from the local computer can gain access, and it must use a password.

Protecting the MySQL Installation Under Windows

The way of proceeding under Windows is very similar to that under Unix/Linux. However, the default configuration under Windows is even more insecure, for which reason a bit more work is necessary.

Furthermore, using the commands mysql and mysqladmin is more complex (in particular, when you wish to pass parameters). The simplest way of proceeding is first to open a command window (START |PROGRAMS |ACCESSORIES |COMMAND PROMPT) and there use cd to make the bin\ directory of the MySQL installation the current directory. (Hint: You will save some typing if you move the directory via Drag&Drop from Explorer into the command window.)

Tip

In the following, for the sake of clarity, we shall always launch mysql anew in order to carry out the individual security measures. If you wish to execute all of the security measures at once, you will, of course, not have to quit mysql each time.

root Password for Local Access: With the following command you can secure root for the local computer with a password (where instead of xxx you provide the password of your choice). This example assumes that MySQL was installed in the directory Q:\Programs\mysql.

Q:\> cd 'Q:\Program \mysql\bin'
Q:\ ... \bin> mysql -u root
Welcome to the MySQL monitor.
mysql> SET PASSWORD FOR root@localhost = PASSWORD('xxx');
Query OK, 0 row affected (0.00 sec)
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.01 sec)

Instead of executing SET PASSWORD in mysql, it must also be possible simply to execute the command mysqladmin -u root -h localhost password xxx in order to change the password for root. However, it appears that this command does not always function reliably under Windows, and often, instead of changing the root password for localhost, changes the root password for access by any number of other computers.

No root Access from External Computers: The following commands prevent root access from external computers.

Q:\ ... \bin> mysql -u root -p
Enter password: xxx
Welcome to the MySQL monitor.
mysql> USE mysql;
mysql> DELETE FROM user WHERE user='root' AND host='%';
Query OK, 1 row affected (0.00 sec)
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.01 sec)

Fewer Privileges for Local Users: In the default setting, local users have almost unrestricted rights, even if they do not register as root. The following command deletes all privileges for nonroot access.

mysql> REVOKE ALL ON *.* FROM ''@localhost;
Query OK, 1 row affected (0.00 sec)
mysql> REVOKE GRANT OPTION ON *.* FROM ''@localhost;
Query OK, 1 row affected (0.00 sec)
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.01 sec)

No Local Access Without a Password: Local users can still access universally visible databases (for example, those whose name begins with test_).

If it is not your wish that anyone with access to the local computer be able to register without a password, you can do without the above command and instead simply forbid local access without a password. There is another reason to recommend this course of action: The entry User = '' and Host = 'localhost' is often unsuspectingly given preference to other entries in the user table (e.g., those with User = 'aname' and User = 'a name' and Host = '%'). By deleting the entry, you avoid possible confusion:

mysql> DELETE FROM user WHERE user='' AND host='localhost';
Query OK, 1 row affected (0.00 sec)
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.01 sec)

From now on, administrative tasks are possible only as root under localhost. Therefore, you must use the commands mysql and mysqladmin with the options -u root and -p.

No Access from External Computers Without a Password: In the default setting, anyone can register with MySQL from an external computer. This access comes with no privileges, but nonetheless it should be prevented. Here are the necessary commands:

mysql> DELETE FROM user WHERE host='%' AND user='';
Query OK, 1 row affected (0.00 sec)
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.01 sec)

WinMySQLadmin Users: If you specified a user name and password at the initial launch of WinMySQLadmin, then under some conditions, the program has an additional user with unrestricted privileges (like those of root). This is particularly the case for older versions of WinMySQLadmin. In the following it is assumed that the user name is namexy.

User namexy is secured by a password, but this password is stored in plain text in the file Windows\my.ini. Anyone who is permitted to work at the computer and knows a bit about MySQL can easily obtain unrestricted access to MySQL. You have three options for closing this security loophole:

  • Delete the user namexy.

  • Delete the password line in Windows\my.ini.

  • Restrict the privileges of this user.

The first two of these options are not ideal, because the program WinMySQLadmin can then be used only with severe restrictions. A workable compromise consists in setting the privileges of namexy in such a way that the most important administrative tasks are possible, but alteration of data is forbidden.

mysql> REVOKE INSERT, UPDATE, DELETE, DROP, FILE, ALTER
     > ON *.* FROM namexy@localhost;
Query OK, 1 row affected (0.00 sec)
mysql> REVOKE GRANT OPTION ON *.* FROM namexy@localhost;
Query OK, 1 row affected (0.00 sec)
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.01 sec)
Remark

From this point on, mysql commands will be shown without the mysql> prompt.

Setting Access Without a Password

Perhaps you were too restrictive in securing MySQL and now would like to provide general access on the local computer (and possibly to a set of particular other computers as well) without the necessity of providing a user name and password. Then simply execute in mysql the following command:

GRANT USAGE ON *.* TO ''@localhost

Perhaps you would like to give the users some more privileges. Then in the command given above replace USAGE by a list containing, for example, SELECT, INSERT, UPDATE, etc. (A list of all possible privileges that are available under MySQL appears in the next section.)

Creating a New Database and User

Among the bread-and-butter tasks of a database administrator is setting up a new database and making it available to a user (who then can insert tables and fill them with data).

This task is easily accomplished. You have merely to execute the following two commands with mysql. The result is the creation of the database forum, to which the user forumadmin has been granted unrestricted access. (If you execute the commands in mysql, do me and yourself a favor and do not forget to terminate the commands with a semicolon.)

CREATE DATABASE forum
GRANT ALL ON forum.* TO forumadmin@localhost IDENTIFIED BY 'xxx'

It is often useful to create a second access to the database, one with fewer privileges (and therefore with fewer security risks):

GRANT SELECT, INSERT, UPDATE, DELETE ON forum.*
  TO forumuser@localhost IDENTIFIED BY 'xxx'

Depending on the application, it can be a good idea to give ordinary users the ability to lock (LOCK ) tables and create temporary tables. This is possible since MySQL 4.0. (In MySQL 3.23, every user who had the right to change tables could execute LOCK.Everyone could create temporary tables who had the right to create any tables at all.)

In the following command, the IDENTIFIED part is omitted. The password of forumuser is then left unchanged:

GRANT LOCK TABLES, CREATE TEMPORARY TABLES ON forum.*
TO forumuser@localhost

Creating New Users

After the database that we have just created, forum, has been in operation for a while, it turns out that another individual, operating from another computer, requires unrestricted access to this database. The following command gives user forumadmin2 on computer uranus.sol full privileges:

GRANT ALL ON forum.* TO forumadmin2@uranus.sol IDENTIFIED BY 'xxx'

If you wish to allow forumadmin2 to sign in from any computer, then the command looks like this:

GRANT ALL ON forum.* TO forumadmin2@'%' IDENTIFIED BY 'xxx'

If you wish to go so far as to allow forumadmin2 access to all databases (and not just forum), the command is as follows:

GRANT ALL ON *.* TO forumadmin2@uranus.sol IDENTIFIED BY 'xxx'

In comparison to root, the only privilege that forumadmin2 does not possess is the right to change access privileges. But a small change remedies the situation:

GRANT ALL ON *.* TO forumadmin2@uranus.sol
IDENTIFIED BY 'xxx' WITH GRANT OPTION

Granting the Right to Create One's Own Database

When there are many users each with his or her own databases (for example, on the system of an Internet service provider), it becomes increasingly burdensome for the administrator to create yet another database for a particular user. In such cases it would be a good idea to give this user the right to create databases. In order to prevent a jungle of databases from appearing with no clue as to which belongs to whom, it is usual practice to allow a user to create only databases that begin with that user's user name. That is, a user named, say, kofler, is allowed to create databases named, say, kofler_test, kofler_forum, kofler1, koflerXy, etc., but no databases whose names do not begin with kofler.

In the following, the way to proceed will be demonstrated for a user with the remarkable user name username. If this user is not yet known to MySQL, then first it must be created:

GRANT USAGE ON *.* TO username@localhost IDENTIFIED BY 'xxx'

Then, sad to tell, you must wrestle with an INSERT command, since the more convenient GRANT command does not, alas, permit wild cards in the specification of database names. Therefore, the necessary changes in the database db must be made directly:

INSERT INTO mysql.db (Host, Db, User, Select_priv, Insert_priv,
  Update_priv, Delete_priv, Create_priv, Drop_priv, Grant_priv,
  References_priv, Index_priv, Alter_priv, Create_tmp_table_priv,
  Lock_tables_priv)
VALUES ('localhost', 'username%', 'username', 'Y', 'Y', 'Y', 'Y',
'Y', 'Y', 'N', 'Y', 'Y',
'Y', 'Y', 'Y')

For these changes to become effective, FLUSH PRIVILEGES must finally be executed.

FLUSH PRIVILEGES

Honey, I Forgot the root Password!

What do you do if you have forgotten the root password for MySQL (and there is no other MySQL user with sufficient administrative privileges and a known password to restore the forgotten password)?

Fear not, for I bring you glad tidings. MySQL has thought about this possibility. The way to proceed is this: Terminate MySQL (that is, the MySQL server mysqld) and then restart it with the option --skip-grant-tables. The result is that the table with access privileges is not loaded. You can now delete the encrypted password for root, terminate MySQL, and then restart without the given option. Now you can give the root user a new password.

The following example is based on MySQL under Linux, although the same procedure is possible under other configurations (though with slight variations). In each case, we assume that you have system administrator privileges on the operating system under which MySQL is running.

The first step is to terminate MySQL:

root# /etc/rc.d/mysql stop

Under Windows you end MySQL in the Service Manager (CONTROL PANEL | ADMINISTRATIVE TOOLS |SERVICES).

In the second step you relaunch mysqld via safe_mysqld (a launch script for mysqld) with the option --skip-grant-tables. The option --user specifies the account under which mysqld should be executed. The option --datadir tells where the MySQL databases can be found. Here the same setting as with a normal MySQL launch should be used. (The precise instruction depends on the system configuration.)

root# startproc /usr/bin/_mysqld_safe --user=mysql \
    --datadir=/var/lib/mysql --skip-grant-tables

Under Windows, execute the following commands for a manual start:

C:\> cd mysql\bin
C:\mysql\bin> mysqld --skip-grant-tables

Now you can use mysql to reset the root password both for the host name localhost and the actual computer name. (Under Windows you must execute mysql in a second command window, since the first window is blocked by mysqld.)

root# mysql -u root
Welcome to MySQL monitor.
mysql> USE mysql;
Database changed.
mysql> UPDATE user SET password=PASSWORD('new password')
> WHERE user='root' AND host='localhost';
Query OK, 1 row affected (0.00 sec)
mysql> UPDATE user SET password=PASSWORD('new password')
> WHERE user='root' AND host='computername';
Query OK, 1 row affected (0.00 sec)

Then, you must restart MySQL so that the access database mysql can again be used. (As long as MySQL is running with --skip-grant-tables, anyone can establish a connection to the server with unlimited privileges!) Under Linux you execute the following commands:

root# /etc/init.d/mysql[d] start
root# /etc/init.d/mysql[d] start

Under Windows, you stop MySQL with the following command:

C:\mysql\bin> mysqladmin shutdown

For a subsequent restart, use either the SERVICES dialog (CONTROL PANEL |ADMINISTRATIVE TOOLS)orWinMySQLadmin.

Caution

The procedures described here can, of course, be used by an attacker who wishes to spy on your data or manipulate it somehow. The only (fortunately, not easily achieved) condition is to acquire Unix/Linux root access (or administrator rights under Windows). This shows how important it is not only to secure MySQL, but also the computer on which MySQL is running. (The topic of security under Unix/Linux and Windows is, like any number of other topics, beyond the scope of this book.)


The Internal Workings of the Access System

You are perhaps, dear reader, sorely tempted to skip over this foundational section, especially since the management of access privileges is rather complicated. However, I strongly suggest that you gird up your loins and make the effort to read through the description of the mysql tables. Regardless of the means that you use for setting up security on your system, it is helpful first to understand the inner workings of the system itself. This holds even for the case that you yourself are not the manager of access privileges, but that they are managed by a system administrator. It is only when you understand how it all works that you will be able to tell your system administrator just what your needs are.

Two-Tiered Access Control

Access control for MySQL databases is managed in two tiers: In the first tier it is merely checked whether the user has the right to establish a connection to MySQL. This is accomplished by the evaluation of three pieces of information: user name, host name, and password.

Only if a connection can be established does the second level of access control come into play, which involves every single database command. For example, if a SELECT is executed, MySQL checks whether the user has access rights to the database, the table, and the column. If an INSERT is executed, then MySQL tests whether the user is permitted to alter the database, the table, and finally the column.

Privileges

How, then, does MySQL manage the information as to which commands can be executed? MySQL uses tables in which are stored privileges. If a user, let us call her athena, has a SELECT privilege for the database owls, then she is permitted to read all the data in owls (but not to change it). If athena has a global SELECT privilege, then it holds for all databases saved under MySQL.

The privileges recognized by MySQL are displayed in Table 9-1. Note that the names in the corresponding columns of mysql tables always end in _priv. The Select privilege is thus stored in the column Select_priv. In part, the column names are abbreviated (e.g., Create_tmp_table for the Create-Temporary-Table privilege).

Table 9-1: MySQL privileges
 | For access to tables
Select | may read data (SELECT command)
Insert | may insert new records (INSERT)
Update | may change existing records (UPDATE)
Delete | may delete existing records (DELETE)
Lock Tables | may block tables (LOCK )
Execute | may execute stored procedures (since MySQL 5.0)
 | For databases, tables, and indexes
Create | may create new databases and tables
Create Temporary Table | may create temporary tables
Alter | may rename and change the structure of tables (ALTER)
Index | may create and delete indexes for tables
References | undocumented; perhaps in the future will allow one to create relations between tables
Drop | may delete existing tables and databases
 | For file access
File | may read and change files in the local file system
 | For MySQL administration
Grant Option | may give other users one's own privileges
Show Databases | may obtain a list of all databases (SHOW DATABASES)
Process | may list MySQL processes of other users (SHOW PROCESSLIST)
Super | may terminate processes of other users (KILL) and execute certain other administrative commands (CHANGE/PURGE MASTER, SET GLOBAL)
Reload | may execute various commands (reload, refresh, flush-xxx)
Replication Client | may obtain information about the participants in a replication system
Replication Slave | may read MySQL server data via replication
Shutdown | may terminate MySQL

Note also that there are quite a few new privileges in MySQL 4.0 that were not present in version 3.23 (e.g., Create Temporary Table, Execute, Lock Tables, Show Databases, Replication Client, Replication Slave, Super), and that many old favorites have been changed (Process).

In the MySQL documentation, you will encounter (for example, in the description of the GRANT command) the privileges All and Usage. All means that all privileges should be granted with the exception of Grant. Usage means that all privileges should be denied.

All and Usage are thus themselves not independent privileges, but an aid in avoiding a listing of privileges in executing the GRANT command.

The meaning of most of the privileges listed in Table 9-1 should be clear without further explanation. For the not-so-clear privileges there will be some explanation in the coming paragraphs. Replication Client and Replication Slave are described in Chapter 10 in the discussion of replication.

Remark

Privileges are given in this book with an intial capital letter followed by lowercase letters in order to distinguish them from their like-named SQL commands (Select privilege and SELECT command). MySQL couldn't care less how you distribute uppercase and lowercase letters.

The Grant Privilege

The Grant privilege indicates that a MySQL user can dispense access privileges. (This is most easily accomplished with the SQL command GRANT, whence the name of the privilege.) However, the ability to dispense privileges is limited to the privileges possessed by the grantor. That is, no user can give privileges to another that he or she does not already possess.

Caution

The Grant Option privilege is an often overlooked security risk. For example, a test database is created to which all members of a team have unrestricted access. To this end, all the privileges in the relevant entry in the db table are set to Y (the db table will be described a bit later).

A perhaps unforeseen consequence is that everyone who has unrestricted access to this test table can give unrestricted access privileges (either to him- or herself or to other MySQL users) for other databases as well!

The File Privilege

MySQL users with the File privilege may use SQL commands for direct access to the file system of the computer on which the MySQL server is running, for example, with the command SELECT ... INTO OUTFILE name or the command LOAD DATA or with the function LOAD_FILE.

In the case of file access, it is necessary, of course, to pay heed to the access privileges of the file system. (Under Unix/Linux the MySQL server normally runs under the mysql account. Therefore, only those files that are readable by the Unix/Linux user mysql can be read.) Nevertheless, the File privilege is often a considerable security risk.

The Privileges PROCESS and SUPER

The Process privilege gives the user the right to determine, using the command SHOW PROCESSLIST, a list of all processes (connections), including those of other users. (One may obtain a list of one's own processes without this privilege.)

The privilege Super permits the user to end both his own and others' processes with KILL. (If the Super privilege has not been granted, then only the current process can be ended.)

The Super privilege also permits the execution of some administrative commands: CHANGE MASTER for executing the client configuration of a replication system, PURGE MASTER to delete binary logging files, and SET GLOBAL for changing global MySQL variables.

Global Privileges Versus Object Privileges

In MySQL privileges can be chosen to be either global or related to a particular object. Global indicates that the privilege is valid for all MySQL objects (that is, for all databases, tables, and columns of a table).

The management of object-related privileges is somewhat more difficult, but it is also more secure. Only thus can you achieve, for example, that a particular MySQL user can alter a particular table, and not all tables managed under MySQL. An assumption in the use of object-related privileges is that the corresponding global privileges are not set. (What is globally allowed cannot be withheld at the object level.)

This hierarchical idea is maintained within the object privileges as well. First it is checked whether access to an entire database is allowed. Only if that is not allowed is it then checked whether access to the entire table named in the SQL command is allowed. Only if that is forbidden is it then checked whether perhaps access to individual columns of the table is allowed.

The mysql Database

It is not surprising that the management by MySQL of access privileges is carried out by means of a database. This database has the name mysql, and it consists of several tables, responsible for various aspects of access privileges.

Remark

It is sometimes not entirely a simple matter to distinguish among the various uses to which the word "MySQL" is put. In this book we attempt to use different type styles to obtain some degree of clarity:

  • MySQL: the database system as a whole.

  • mysqld : theMySQLserver (MySQL daemon).

  • mysql : theMySQLmonitor (a type of command interpreter).

  • mysql : the database for managing MySQL access privileges.

The Tables of the Database mysql

The database mysql contains six tables (five in earlier versions) of which five are for managing access privileges. These five tables are often referred to as grant tables. The following list provides an overview of the tasks of these six tables:

  • user controls who (user name) can access MySQL from which computer (host name). This table also contains global privileges.

  • db specifies which user can access which databases.

  • host extends the db table with information on the permissible host names (those that are not present in db).

  • tables_priv specifies who can access which tables of a database.

  • columns_priv specifies who can access which columns of a table.

  • func enables the management of UDFs (user-defined functions); this is still undocumented.

    Remark

    When you newly install MySQL, the default values of user and db depend on the operating system. The contents of these two tables are displayed in Tables 9-3, 9-4, and 9-6, which appear later. The effect of these settings was discussed earlier. (The tables host, tables_priv,and columns_priv start off life empty.)

    In MySQL's two-tiered access system, the table user is solely responsible for the first level (that is, for the connection to MySQL). The user table contains all global privileges.

    For the second tier (that is, access to specific objects: databases, tables, and columns) it is the tables db, host, tables_priv, and columns_priv that are responsible, in addition to user.

    The tables db, host, tables_priv, and columns_priv come into play in this order when the privileges for their respective tiers are set to N (which stands for "No"). In other words, if the SELECT privilege is granted to a user in user, then the other four tables will not be consulted in checking the permissibility of a SELECT command executed by that user.

    The result is that if fine distinctions in access privileges are to be made, the global privileges in the user table must be set to N.

An Example

Before we attempt to describe the individual tables in detail, we present an example: Let us suppose that a particular MySQL user, let us call him zeus, is to be allowed to read a particular column of a particular table, and nothing else. Then zeus (user name, host name, password) must be entered in the user table. There all of zeus's global privileges will be set to N.

Furthermore, zeus (user name, host name) must be registered in the columns_priv table as well. There it must also be specified which column zeus is permitted to access (database name, table name, column name). Finally, there the Select privilege (and only this privilege) must be activated. In the tables db, host, and tables_priv no entries for zeus are necessary.

The Table user

Table 9-2: Structure of the user table
Field | Type | Null | Default
Host | char(60) | No |
User | char(16) | No |
Password | char(16) (char(45) since MySQL 4.1) | No |
Select_priv | enum('N', 'Y') | No | N
Insert_priv | enum('N', 'Y') | No | N
Update_priv | enum('N', 'Y') | No | N
Delete_priv | enum('N', 'Y') | No | N
Create_priv | enum('N', 'Y') | No | N
Drop_priv | enum('N', 'Y') | No | N
Reload_priv | enum('N', 'Y') | No | N
Shutdown_priv | enum('N', 'Y') | No | N
Process_priv | enum('N', 'Y') | No | N
File_priv | enum('N', 'Y') | No | N
Grant_priv | enum('N', 'Y') | No | N
References_priv | enum('N', 'Y') | No | N
Index_priv | enum('N', 'Y') | No | N
Alter_priv | enum('N', 'Y') | No | N
Show_db_priv | enum('N', 'Y') | No | N
Super_priv | enum('N', 'Y') | No | N
Create_tmp_table_priv | enum('N', 'Y') | No | N
Lock_tables_priv | enum('N', 'Y') | No | N
Execute_priv | enum('N', 'Y') | No | N
Repl_slave_priv | enum('N', 'Y') | No | N
Repl_client_priv | enum('N', 'Y') | No | N
ssl_type | enum('', 'ANY', 'X509', 'SPECIFIED') | No | 0
ssl_cipher | blob | No | 0
x509_issuer | blob | No | 0
x509_subject | blob | No | 0
max_questions | int | No | 0
max_updates | int | No | 0
max_connections | int | No | 0

The user table (see Table 9-2) fulfills three tasks:

  • This table alone regulates who has any access at all to MySQL.

  • global privileges can be granted through this table. Note that the column names of the table differ somewhat from those of the privileges (e.g., the Create_tmp_table column for the Create Temporary Table privilege).

  • Since MySQL 4.0 the table contains several new columns for encrypted access via SSL (secure socket layer), for identity control in accordance with the X509 standard and for managing such control, and for the number of database connections, that is, the maximal number of updates and queries permitted to be executed per hour.

Access Control (User, Host, and Password Columns)

For control over who can connect to MySQL there are three necessary identifiers—as we have mentioned already several times—that must be evaluated: user name, host name, and password. Here we shall say a few words about how this information is stored in the fields User, Host, and Password.

  • User Name: Access control is case-sensitive. In the User field no wild cards are allowed. If the User field is empty, then any user name is permitted for accessing MySQL. Then the user is considered to be an anonymous user. This means that in the second tier of access control the actual user name is not used, but rather an empty character string.

  • Password: The password must be stored in the column Password and there be encrypted with the SQL function PASSWORD. It is not possible to store a password in plain text. Furthermore, wild cards are not allowed. If the password field remains empty, then a connection can be made with no password whatsoever. (An empty password field thus does not mean that an arbitrary password can be given.)

    With MySQL 4.1 the encryption of the Password columns has changed. The encrypted code of the password has been increased from 16 to 45 characters. Be thou once more warned: For reasons of security you should never use as your MySQL password a password that you use to gain admittance to the operating system.

  • Host Name: The host name can be given either as a name or as an IP number. Here the wild cards _ (an arbitrary character) or % (zero or more characters) are permitted, for example, 192.168.37.% or %.myfavoriteenterprise.com. In the case of IP numbers, the forms (which are equivalent to those of the first example) 192.168.37.0/255.255.255.0 and 192.168.37.0/24 are permitted. If the host name is given simply as the character %, a connection can be made from any computer. To enable access to users on the local computer, specify the host name localhost.As with the user name, the host name is case-sensitive.

    To enable access on the local computer, give localhost as the host name. When Unix/Linux users wish to obtain local access via TCP/IP (important for Java applications), then the actual computer name must be used as the host name. If both forms of access are to be enabled, then the user must be defined twice, once with the host name localhost, and once with computername.

    Pointer

    Normally, the host name must be given together with the domain name. Thus if the computer saturn.sol wants to gain access from the network *.sol, the column Host in the user table must contain the character string saturn.sol and not just saturn. Depending on your network configuration and that of the existing domain name server, it could be that one has to specify the host name without the domain name. Further details on this and other picky details will be discussed further along in this chapter.

The Order of Evaluation in the user Table

It often happens that in making a connection to MySQL several records match the given login data. If, for example, you log in as root from the local computer under the default settings of the user table for Windows, then all four entries would match. MySQL then decides in favor of the most nearly exact description (thus in this case for root/localhost and not for ''/%).). The reason for this is that with an exact match in the login one can expect more in the way of access privileges.

In order to speed up the selection of the correct record, the user table is sorted internally by MySQL. Here the field Host serves as the first sort criterion, and User as the second. In sorting, those character strings without the wild cards _ and % are given priority. Empty character strings and the character string % come in last in the sort order. Tables 9-3 and 9-4 show the default settings for the user table in their respective internal sort orders.

Table 9-3: Default setting for the user table under Unix/Linux
Host | User | Password | Select_priv to Alter_priv | Super_priv to Repl_client_priv | ssl_xxx, x509_xxx | max_xxx
localhost | root |  | Y | Y | NULL | 0
computer name | root |  | Y | Y | NULL | 0
localhost |  |  | N | N | NULL | 0
computer name |  |  | N | N | NULL | 0
Table 9-4: Default setting for the user table under Windows
Host | User | Password | Select_priv to Alter_priv | Super_priv to Repl_client_priv | ssl_xxx, x509_xxx | max_xxx
localhost | root |  | Y | Y | NULL | 0
localhost |  |  | Y | N | NULL | 0
% | root |  | Y | Y | NULL | 0
% |  |  | N | N | NULL | 0
Caution

The order in which the entries in the user table are evaluated often leads to unexpected problems. Suppose that the MySQL default settings hold for user privileges and that you add a new user with Host = '%' and User = 'peter' in the user table. If peter now attempts to register from the local computer (that is, from localhost), it is not the entry %/'peter' that comes into play, but the entry from the default setting Host = 'localhost' and User = ''. The reason is that MySQL gives precedence to entries with a unique hostname (Host = 'localhost')overthosewitha wild card (Host='%'). The easiest and surest way out of this difficulty is to delete the entry Host = 'localhost' and User = '' from the user table.

Privileges

In addition to MySQL access control, the user table also has the responsibility for global privileges. To this end there are currently 21 fields that take part, which can be set to Y or N.Since Y stands for "yes," this setting means that the associated privilege is set globally for the MySQL user (for all databases, tables, and columns). Contrariwise, N means "no" (or non in French, nein in German, nyet in Russian; you get the idea), and so such a setting means that the operation in question is not allowed globally, and therefore the tables db, host, tables_priv, and columns_priv will be consulted for object-specific privileges.

SSL Encryption and X509 Identification

To make communication between client and server particularly secure, the X509 standard can be used for user identification, and data transfer can be effected with SSL (secure socket layer) encryption. This requires a special compiled version of the MySQL server such as MySQL Max. (The SSL functions are not currently integrated into MySQL Standard.) Whether your version of MySQL supports SSL can be determined with the SQL command SHOW VARIABLES LIKE 'have_openssl'.

Note that the encryption of data affects the speed of the MySQL server and makes sense only if data transfer is taking place over the Internet, that is, if the MySQL server and the client are running on different computers.

The encryption of a local connection (if, for example, PHP/Apache accesses a MySQL server that is running on the same computer) does not increase the security of the application. In this case, the security is much more greatly determined by how communication between Apache and the web user takes place (e.g., encryption via HTTPS).

Pointer

This book does not go into details of the configuration and application of SSL and X509. More extensive information can be found in the MySQL book of Paul DuBois (second edition) and the MySQL documentation on the Internet:

http://www.mysql.com/doc/en/Secure_connections.html

Limitation of MySQL

With the three columns max_questions, max_updates, max_connections you can specify how many queries (SELECT commands) and data changes (INSERT and UPDATE) are permitted to be executed per hour or how many connections established. The default setting 0 means that there are no restrictions.

Default Setting

As we described in the introduction to this chapter, after installation of MySQL there are various default settings of the user table that depend on the particular operating system. Tables 9-3 and 9-4 give these settings (for MySQL version 4.0.n).

The user.Host Column

As briefly mentioned in the previous section, the Host column of the user table specifies from which computer a particular user is to obtain access to the MySQL server. The computer name can be given with the wild card % or by an IP number.

What looks simple at first glance is, alas, in practice a source of numerous problems. Therefore, this section contains various pieces of background information and some tips for solving connection problems.

Host Name with or without Domain Name?

When you insert a new entry in the user table, the question arises whether the host name should appear with or without the domain name, that is, for example, uranus or uranus.sol. Unfortunately, there is no generally valid answer to this question. You must simply experiment and find out. However, I have made the discovery that the probability of success is higher if you include the domain name. (The resolution of network names depends on the network configuration and, to the extent available, the configuration of the domain name server, thus on factors that lie outside of MySQL.)

Based on the default configuration, under Unix/Linux the host name, that is, the column user.Host, is entered without domain name (see Table 9-3). This is due to the script mysql_install_db, which is executed during the installation of the MySQL server. If MySQL evaluates the complete network name in establishing a connection, then a local connection under TCP/IP is impossible. The problem is that mysql -u root is functioning, but mysql -u root -h computername is not.

In this case, you have to change the domain name explicitly. The necessary commands look like those below (of course, you will have to change uranus.sol and uranus to the names that apply in your situation):

root# mysql -u root
mysql> USE mysql;
mysql> UPDATE user SET host="uranus.sol" WHERE host="Uranus";
mysql> FLUSH PRIVILEGES;

computerame Versus localhost Under Unix/Linux

When the MySQL server is executed under Unix/Linux, the default security settings provide two entries for local access to the server: localhost and computername (see Table 9-3). The reason for this dualism is that connections between the client program and the MySQL server can take place in two ways (only when client and server run on the same computer): over a socket file or via the network protocol TCP/IP.

  • Socket file: The use of a socket file avoids the overhead of TCP/IP. For this reason, most MySQL clients that access MySQL from the local computer use this variant. In checking access privileges, the entry localhost is used.

    The socket file generally has the name var/lib/mysql/mysql.sock. The name can be changed in /etc/my.cnf with the option socket.

  • TCP/IP: However, there are clients that do not support sockets. (This holds in particular for all clients programmed in Java.) In this case, TCP/IP is used even for connections to a local server. In checking access privileges, now the entry computername is used. For this reason it is to be recommended that for every localhost user an equivalent user be created for whom the name of the local computer is specified as host name.

If you launch the program mysql on the same computer as the MySQL server, you can test both connection options: If you do not specify a host name or if you give localhost as host name, the connection succeeds over the socket file (first and second commands below). If, on the other hand, you specify the actual computer name or the IP address, the connection proceeds over TCP/IP (third through sixth commands). You can use the command status to determine how the connection actually is made:

linux:˜ $ mysql                    -u root -p    (1) via Socket-Datei
linux:˜ $ mysql -h localhost       -u root -p    (2) via Socket-Datei
linux:˜ $ mysql -h uranus          -u root -p    (3) via TCP/IP
linux:˜ $ mysql -h uranus.sol      -u root -p    (4) via TCP/IP
linux:˜ $ mysql -h 127.0.0.1       -u root -p    (5) via TCP/IP
linux:˜ $ mysql -h 192.168.0.2     -u root -p    (6) via TCP/IP
Welcome to the MySQL monitor. ...
mysql> status
...
Current user:
   root@uranus.sol
SSL:                 Not in use
Current pager:       less
Using outfile:       ''
Server version:      4.0.4-beta-log
Protocol version:    10
Connection:
   uranus via TCP/IP
TCP port:
   3306
...

Remark

When client and server run on different computers, the connection always runs via TCP/IP, and there is no issue of duality.

Under Windows NT/2000/XP there are two possibilities for a local connection: Communication can take place over TCP/IP or over a named pipe (mysql option --pipe). However, this works only if the MySQL server was compiled with named pipe support, which is not the default. In contrast to the situation under Unix/Linux, the same user entry in the user table is used. The host name localhost is used for all local connections, regardless of how the connection is made.

localhost Problems with Red Hat Linux

With Red Hat Linux it can be necessary, depending on the configuration, to enter localhost.localdomain in the Host column for local TCP/IP connections (instead of the otherwise usual computer name). The reason is that with Red Hat, the IP address 127.0.0.1 is linked to the name localhost.localdomain (file /etc/hosts).

localhost Problems with SuSE Linux

With SuSE as well there can be problems with localhost. The reason is that after the execution of a network configuration, the line 127.0.0.2 computername is entered in /etc/hosts. This line can cause confusion if computername is associated with a different IP address. Such problems generally occur only when a local TCP/IP connection is sought. There are several options for alleviating this problem:

  • As a first attempt, you should try to specify the complete computer name (including domain name) in the Host table. At least in my configuration, this was sufficient for establishing a local TCP/IP connection in spite of the 127.0.0.2 line in /etc/hosts.

  • If that doesn't work, you can enter the IP number 127.0.0.2 in the Host column instead of the computer name. This should usually work, but it actually only alleviates the symptom, and does not get at the cause.

  • A third variant is to specify the complete computer name in the Host column and to comment out the 127.0.0.2 line in /etc/hosts (by prefixing a # sign). However, then you must also set the variable CHECK_ETC_HOSTS in /etc/sysconfig/suseconfig to no. Otherwise, YaST undoes your change in /etc/hosts at the next opportunity.

    Then you must restart the network and the MySQL server, so that the changes can take effect and so that the MySQL server will no longer use temporary storage for name resolution:

    root# /etc/init.d/network restart
root# /etc/init.d/mysql stop
root# /etc/init.d/mysql start

The main disadvantage of this variant is that there can now be compatibility problems with other SuSE packages that may depend on the SuSE standard configuration.

Establishing the Host Name and IP Address Under Unix/Linux

You can determine the current computer name with the command hostname. If you then pass this computer name to the command host, the complete computer name (including domain name) and associated IP address will be returned. Conversely, host can return a computer name given an IP address:

linux:~ $ /hostname
uranus
linux:~ $ /host uranus
uranus.sol. has address 192.168.0.2 linux:~ $ /host uranus.sol
uranus.sol. has address 192.168.0.2 linux:~ $ /host 192.168.0.2
2.0.168.192.in-addr.arpa. domain name pointer uranus.sol.

In addition to host, you can also use the command resolveip, which is installed with MySQL. The MySQL server uses the same algorithm as resolveip in resolving network names and IP addresses. If resolveip and host give different results, there is a problem in the configuration. The most likely source of error is the file /etc/hosts or an incorrect configuration of the domain name server (DNS) if one is used in the local network:

linux:~ $ /resolveip uranus
IP address of uranus is 192.168.0.2
linux:~ $ /resolveip uranus.sol
IP address of uranus is 192.168.0.2
linux:~ $ /resolveip 192.168.0.2
Host name of 192.168.0.2 is uranus.sol
Tip

The MySQL server takes care by default of the ordering of IP numbers and host names (so that repeated accesses can take place as rapidly as possible). If you change the network configuration, you must delete this temporary storage:

root# mysqladmin flush-hosts
Remark

Under Windows, the commands hosts and resolveip are not available. However, you can determine the computer name as under Unix/Linux with the command hostname.

Host Name or IP Address?

In principle, in the Host column you can specify the computer's IP address instead of the computer name. The advantage of IP numbers is that the numerous sources of error in name resolution no longer exist. However, this advantage is paid for in the drawback that in many networks, IP numbers frequently change (in some circumstances at each restart if the IP number is assigned dynamically by a DHCP server). Therefore, controlling access on the basis of IP addresses can succeed only if the management of IP addresses in your network is organized relatively strictly.

Remark

If there exist equivalent entries in the user table that differ only in that one has an IP number and the other an associated computer name, then the entry with the computer name takes precedence.

The db and host Tables

The db Table

The db table (see Table 9-5) contains information on which databases a particular user is permitted to read, edit, and delete. The function of this table is easy to understand: If user u located at computer h wishes to access database d by executing a SELECT command and does not possess a global Select privilege, then MySQL pores over the db table and looks for the first User/Host/Db entry for the triple u/h/d. (As we saw in the case of the user table, uniquely identifying entries take precedence over those with wild cards. The entries are case-sensitive.) If a matching entry is found, then all that must be checked is whether the column Select_priv contains the value Y.

Table 9-5: Structure of the db table
Field | Type | Null | Default
Host | char(60) | No |
Db | char(64) | No |
User | char(16) | No |
Select_priv | enum('N', 'Y') | No | N
Insert_priv | enum('N', 'Y') | No | N
Update_priv | enum('N', 'Y') | No | N
Delete_priv | enum('N', 'Y') | No | N
Create_priv | enum('N', 'Y') | No | N
Drop_priv | enum('N', 'Y') | No | N
Grant_priv | enum('N', 'Y') | No | N
References_priv | enum('N', 'Y') | No | N
Index_priv | enum('N', 'Y') | No | N
Alter_priv | enum('N', 'Y') | No | N
Create_tmp_table_priv | enum('N', 'Y') | No | N
Lock_tables_priv | enum('N', 'Y') | No | N
Remark

In order for security control to be executed quickly, MySQL maintains various mysql tables presorted in RAM. However, this has the consequence that direct changes in these tables become active only when MySQL is ordered to reread these tables with FLUSH PRIVILEGES or mysqladmin reload.

Practically the same rules as in the user table hold for the settings of User and Host in the db table. The only exception relates to the Host column: If this remains empty, then MySQL evaluates the host table as well (see the heading "The host Table").

Default Setting

After installation of MySQL, all users (those who are able to access MySQL) are permitted to set up, edit, and delete databases whose names begin with test_. (Under Windows, the underscore in the database name is not required.) This default setting is supposed to allow a nonbureaucratic testing of MySQL without the beleaguered system administrator having to set up such databases.

An obvious problem with this arrangement is that anyone with access to MySQL has the right to create a database and stuff it with data and then stuff it some more until the computer's hard drive is full.

Table 9-6 shows the default setting for the db table. Please note in particular that for test databases all privileges except for Grant, Create Temporary Tables, and Lock Tables are set. Grant='n' is important, so that privileges from the test databases cannot be transferred to other databases.

Table 9-6: Default setting of the db table under Unix/Linux
Host | Db | User | Select_priv, Insert_priv, etc. | Grant_priv, Create_tmp_table_priv, Lock_tables_priv
% | test_% |  | Y | N
Remark

Have you understood the privileges system of MySQL? Then see whether you can answer the following question:

Can a MySQL user create and edit a test database even if all global privileges are set to N?

The answer is yes, of course. If all the global privileges are set to N, then the object-specific privileges are consulted (where every database that begins with test is considered an object). This is precisely the concept of the privilege system of MySQL.

The host Table

The host table (see Table 9-7) is an extension of the db table if the latter's Host field is empty. In this case, entries for the database in question are sought in the host table. If an entry that matches the computer name is found there, the privilege settings in db and host are joined with a logical AND (that is, privileges must be granted in both tables).

Table 9-7: Structure of the host table
Field | Type | Null | Default
Host | char(60) | No |
Db | char(64) | No |
Select_priv | enum('N', 'Y') | No | N
Insert_priv. . . | enum('N', 'Y') | No | N
Lock_tables_priv | enum('N', 'Y') | No | N

The host table is brought into play relatively rarely. (As a rule, the settings of the db table meet all requirements.) This is also expressed in the fact that the commands GRANT and REVOKE, introduced below, do not affect the host table. Thus the host table is empty in the default setting.

The tables_priv and columns_priv Tables

With the tables tables_priv (see Table 9-8) and columns_priv (see Table 9-9) privileges can be set for individual tables and columns. For Host and User the same rules hold as for the user table. In the fields Db, Table_name, and Column_name,on the other hand, no wild cards are permitted. All the fields except Column_name are case-sensitive.

Table 9-8: Structure of the tables_priv table
Field | Type | Null | Default
Host | char(60) | No |
Db | char(64) | No |
User | char(16) | No |
Table_name | char(64) | No |
Grantor | char(77) | No |
Timestamp | timestamp(14) | Yes |
Table_priv | set1 | No |
Column_priv | set2 | No |
Table 9-9: Structure of the columns_priv table
Field | Type | Null | Default
Host | char(60) | No |
Db | char(64) | No |
User | char(16) | No |
Table_name | char(64) | No |
Column_name | char(64) | No |
Timestamp | timestamp(14) | Yes |
Column_priv | set2 | No |

In the undocumented column Grantor is stored information as to who has granted the access rights (for example, root@localhost).

Unlike the tables described above, these tables have their privileges managed by two sets. (Sets are a peculiarity of MySQL; see also Chapter 5. With sets an arbitrary combination of all character strings specified by set definitions can be stored.) The two sets for tables_priv and columns_priv look like this:

set1: SET('Select', 'Insert', 'Update', 'Delete', 'Create', 'Drop',
         'Grant', 'References', 'Index', 'Alter')
set2: SET('Select', 'Insert', 'Update', 'References')

Here set2 contains only those privileges that hold for individual columns. (The Delete privilege is missing, because only an entire data record can be deleted, not an individual field of a record. This field can, of course, be set to NULL, 0, or '', but for that the Update privilege suffices.)

It is my sad duty to report that it is not documented as to when in the table tables_priv the field Table_priv is to be used and when the field Column_priv. If one considers how the fields are changed by the SQL command GRANT, one may venture the following hypothesis:

  • Privileges that relate to the entire table are stored in the field Table_priv.

  • Privileges that relate to individual columns are stored in the field Column_priv, as well as in the tables_priv and columns_priv tables. The tables_priv table is to acertain extent a conglomeration of all privileges for which there are additional details in the columns_priv table (perhaps divided over several data records). The reason for this is probably the idea that evaluating both tables leads to simplification or increased speed for MySQL.

If you are in doubt, use the commands GRANT and REVOKE for setting column privileges. Have a look at the MySQL documentation. Perhaps since this was written the Columns_priv field has received a better treatment.

The tables tables_priv and columns_priv are empty in the default setting.


Tools for Setting Access Privileges

One can edit the tables of a database (assuming, of course, that you have the appropriate access privileges) with the usual SQL commands INSERT, UPDATE, and DELETE. However, that is a tiring and error-prone occupation. It is much more convenient to use the commands GRANT and REVOKE, which are the centerpiece of this section. Further alternatives for particular tasks are the MySQL tool mysqladmin and the Perl script mysql_setpermission.

Caution

MySQL maintains, for reasons of speed optimization, copies of the mysql tables in RAM. Direct changes to the tables are effective only if they are explicitly reread by MySQL via the SQL command FLUSH PRIVILEGES or the external program mysqladmin reload. (With GRANT and REVOKE this rereading takes place automatically.)

Changing Access Privileges with GRANT and REVOKE

The syntax of the GRANT and REVOKE commands, in simplified form, is as follows:

GRANT privileges
ON [database.]table
TO user@host [IDENTIFIED BY 'password']
[WITH GRANT OPTION]
REVOKE privileges
ON [database.]table
FROM user@host

If you wish to change the access privileges for all the tables of a database, the correct form to use is ON database.*. If you wish to alter global privileges, then specify ON *.*.Itis not allowed to use wild cards in database names.

For user you can specify '' to indicate all users on a particular computer (for example, ''@computername). On the other hand, for host you must use '%' (for example, username@'%').

Depending on their function, these commands change the mysql tables user, db, tables_priv, and columns_priv.(The host table remains untouched.)

Pointer

The complete syntax of GRANT and REVOKE is given in Chapter 18. However, you can get a fairly good feel for these commands from the following examples.

Registering New Users

All users with the computer name *.myorganization.com are permitted to link to MySQL if they know the password xxx. The privilege USAGE means that all global privileges have been set to N. The users thereby at first have no privileges whatsoever (to the extent that so far no individual databases, tables, or columns have been made accessible to all users who can log into MySQL):

GRANT USAGE ON *.* TO ''@'%.myorganization.com' IDENTIFIED BY 'xxx'

The following command gives the user admin on the local computer unrestricted privileges. All privileges (including Grant)are set:

GRANT ALL ON *.* TO admin@localhost IDENTIFIED BY 'xxx'
WITH GRANT OPTION

Enabling Access to a Database

The following command gives the user peter on the local computer the right to read and alter data in all tables of the database mylibrary. If peter@localhost is unknown to the user table of the mysql database, then this name is added without a password. (If there is already a peter@localhost, then the password is not changed.)

GRANT SELECT, INSERT, UPDATE, DELETE
ON library.* TO peter@localhost

If you wish to add to peter's privileges the right to lock tables and create temporary tables (which is useful in many applications), the command looks like this:

GRANT SELECT, INSERT, UPDATE, DELETE, CREATE TEMPORARY TABLES, LOCK TABLES
ON library.* TO peter@localhost

Prohibiting Changes in a Database

The next command takes away from peter the right to make changes to mylibrary, but peter retains the right to read the database using SELECT (assuming that the command of the previous example was just executed).

REVOKE INSERT, UPDATE, DELETE
ON mylibrary.* FROM peter@localhost

Enabling Access to Tables

With the following command the user kahlila on the local computer is given the right to read data from the table authors in the database mylibrary (but not to alter it):

GRANT SELECT ON mylibrary.authors TO kahlila@localhost

Enabling Access to Individual Columns

The access privileges for katherine are more restrictive than those for kahlila: She is permitted only to read the columns title and subtitle of the table books in the database mylibrary.

GRANT SELECT(title, subtitle) ON mylibrary.books TO katherine@localhost

Granting Database Access to All Local Users

All users on the local computer can read and edit data in the mp3 database:

GRANT SELECT, INSERT, DELETE, UPDATE ON mp3.* TO ''@localhost
Pointer

An additional example for changing access privileges can be found in this chapter under the heading "First Aid."

Viewing Access Privileges with SHOW GRANTS

If you have lost track of which privileges a particular user has, the command SHOW GRANTS is just what you need:

SHOW GRANTS FOR peter@localhost;
    Grants for peter@localhost:
    GRANT SELECT ON mylibrary.* TO 'peter'@'localhost'
1  row in set (0.00 sec)
SHOW GRANTS FOR testuser@localhost
    GRANT USAGE ON *.* TO 'testuser'@'localhost'
        IDENTIFIED BY PASSWORD '663c5dd53dae4ed0'
    GRANT SELECT, INSERT, UPDATE, DELETE,
        CREATE TEMPORARY TABLES, LOCK TABLES
        ON 'myforum'.* TO 'ptestuser'@'localhost'

Changing a Password with mysqladmin

The program mysqladmin carries out various administrative tasks (see also Chapter 10). Although this program does not offer any immediate assistance in managing access privileges, it does offer two applications that seem to fit into this chapter.

Changing a Password

You can use GRANT to change the password of a previously registered user. However, GRANT can be used only when at the same time access privileges are to be changed. If all you want to do is to change a password, then mysqladmin is a simpler alternative:

> mysqladmin -u peter -p password newPW
Enter password: oldPW

The above command changes the password for the user peter on the computer localhost. Please note that the new password is passed as a parameter, while the old password is entered on request. (This order, first the new and then the old, is rather unusual.)

Caution

Under Windows, mysqladmin frequently causes problems, because it does not correctly interpret the local computer name localhost. Often, the result is the error message Can't find any matching row in the user table or Access denied for user: 'some-user@unknown' to database 'mysql'.

It is even more dangerous if mysqladmin is executed without any error message, and insead of changing the passward for username@localhost, changes it for username@'%'.(This can happen only if there exists such a user.)

The solution is to use the program mysql and change the password with the following command:

SET PASSWORD FOR username@localhost = PASSWORD('xxx')

Reading in New mysql Tables

For speed enhancement, MySQL loads mysql into RAM at startup. The disadvantage of this is the following: If you change mysql directly using the INSERT, UPDATE, or DELETE commands, MySQL simply ignores these changes until you execute either the SQL command FLUSH PRIVILEGES or mysqladmin reload:

> mysqladmin -u root -p reload
Enter password: xxx

It is not necessary to use mysqladmin reload if you wish to change access privileges with GRANT or REVOKE. In those cases the altered tables are automatically read into RAM.

Viewing Access Privileges with mysqlaccess

Here we note that mysqlaccess is a rather old Perl script (last change in December 2000) that summarizes the access privileges of MySQL users. Thus mysqlaccess has a function similar to that of SHOW GRANTS, but with more features. It gives information as to which entries in the mysql tables are used for access, and it indicates security loopholes. This script is therefore useful if you need to get an overview of MySQL access privileges.

Remark

Information on using mysqlaccess can be obtained by executing the script with the option --help or --howto. The script can be used only under Unix/Linux, because in addition to the Perl interpreter, Unix-specific programs such as diff are necessary.

Our first example shows how peter is able to access the database mylibrary:

user@linux:~ > mysqlaccess peter library
Password for MySQL superuser root: xxx

Sele | Inse | Upda | Dele | . . . | Alte | Host,User,DB
Y | N | N | N | . . . | N | localhost,peter,library

In our second example it is determined who (user name *) is permitted to access the mysql database:

user@linux:~ > mysqlaccess '*' mysql
Password for MySQL superuser root: xxx

Sele | Inse | Upda | Dele | . . . | Alte | Host,User,DB
Y | Y | Y | Y | . . . | Y | localhost,root,mysql
N | N | N | N | . . . | N | localhost,peter,mysql
N | N | N | N | . . . | N | localhost,ANY_NEW_USER,mysql

Changing Access Privileges with mysql_setpermission

If you find the commands GRANT and REVOKE too burdensome for standard tasks (such as giving a user access to a database), you may instead use the Perl script mysql_setpermission. The script is very simple to use, though the number of possible operations is small. (In particular, it is not possible to restrict privileges that have already been granted.)

Remark

The script mysql_setpermission assumes that Perl is installed on your computer together with the DBI and MySQL modules. The Windows version of the program can be found after a complete installation of MySQL in the directory mysql\scripts.

The following example shows how with mysql_setpermission the database webforum is set up for the user forumadmin.As always, inputs are shown in boldface.

user@linux:~ > mysql_setpermission -u root
Password for user root to connect to MySQL: xxxx
What would you like to do:
 1. Set password for a user.
 2. Add a database + user privilege for that database.
  - user can do all except all admin functions
 3. Add user privilege for an existing database.
  - user can do all except all admin functions
 4. Add user privilege for an existing database.
  - user can do all except all admin functions + no create/drop
 5. Add user privilege for an existing database.
  - user can do only selects (no update/delete/insert etc.)
 0. exit this program
Make your choice [1,2,3,4,5,0]: 2
Which database would you like to add: webforum
The new database webforum will be created
What user name is to be created: forumadmin
user name = forumadmin
Would you like to set a password for [y/n]: y
What password do you want to specify for: newPW
Type the password again: newPW
We now need to know from what host(s) the user will connect.
Keep in mind that % means 'from any host' ...
The host please: localhost
Would you like to add another host [yes/no]: n
Okay we keep it with this ...
The following host(s) will be used: localhost.
That was it ... here is an overview of what you gave to me:
The database name       : webforum
The user name           : forumadmin
The host(s)             : localhost
Are you pretty sure you would like to implement this [yes/no]: y
Okay ... let's go then ...
Everything is inserted and mysql privileges have been reloaded.

With mysqlaccess you can quickly check which privileges mysql_setpermission has granted (in our example, all except Reload, Shutdown, Process, and File). The output of mysqlaccess is here shown in greatly compressed form over three lines:

user@linux:~ > mysqlaccess forumadmin webforum
Password for MySQL superuser root: xxx
Sele | Inse | Upda | Dele | Crea | Drop | Relo | Shut | Proc | File | Gran
Y | Y | Y | Y | Y | Y | N | N | N | N | Y
Refe | Inde | Alte | Show | Supe | Crea | Lock | Exec | RepS | RepC
Y | Y | Y | N | N | N | N | N | N | N
SslT | SslC | X509 | X509 | MaxQ | MaxU | MaxC | host | User | DB
? | ? | ? | ? | 0 | 0 | 0 | localhost | forumadmin | webforum


Problems with Establishing a Connection

Tip

If you experience connection problems, regardless of what programming language you are using, you should first perform a test interactively with the program mysql to determine whether you can establish a connection. Only when that succeeds is it worthwhile pursuing a search for the cause of the error in the program that you are using or developing.
Warning

Changes to access privileges or to the mysql database become effective only after FLUSH PRIVILEGES is executed (or after a restart of the MySQL server). Many administration programs with dialogs for user management do not automatically execute FLUSH PRIVILEGES.

Therefore, if you have changed access privileges but see no result, execute in mysql as root the command FLUSH PRIVILEGES (or the equivalent mysqladmin flush-privileges).

In the case of an existing connection, changes made may take effect in part only after a new login, despite the execution of FLUSH PRIVILEGES. The following rules are in effect:

  • Changed global privileges will go into effect only after a new connection is made.

  • Changed database privileges go into effect only after the command USE table is executed.

  • Changed table and column privileges go into effect at the next SQL command.

The following list gives the typical causes of problems in establishing connections. Note that a particular error message can arise from one of several different causes.

  • The MySQL server does not run: If you attempt a connection with mysql, you obtain error 2002 (Can't connect to MySQL server on 'hostname') or error 2003 (Can't connect to local MySQL server through socket /var/lib/mysql/mysql.sock).

    Under Windows, you can tell whether the server is running by looking in the task manager, while under Unix you use the command ps | grep -i mysql. As result, a list of processes should appear (since the server divides itself into a number of processes for reasons of efficiency). If that does not occur, then the server must be started (under Linux with the command /etc/init.d/mysql[d] start).

  • The client program does not find the socket file: Under Unix/Linux, communication takes place mostly over a socket file if server and client are running on the same computer. For this to function, both programs must agree on the location of this file. When problems arise, you should ensure that there is an entry socket=filename in the configuration file /etc/my.cnf in the section [client], where filename specifies the actual location of the socket file. Normally, this file has the name /var/lib/mysql/mysql.sock.

  • The network connection between client and server is broken: If your program is running on a different computer from that of the MySQL server, execute on the client computer the command ping serverhostname to test whether a connection to the server computer exists. If that is not the case, you must first repair the network configuration.

  • MySQL accepts no connections over the network (over TCP/IP): This can be achieved with the option –skip-networking or a corresponding setting in my.cnf. This setting is often chosen to give MySQL maximum security. A database connection is then available only from the local computer and only via a socket file.

    The problem is generally recognizable from error 2003 (Can't connect to MySQL server). A solution is to remove the option my.cnf from the start script.

  • MySQL accepts no connections from your computer: This problem generally arises when the MySQL server is running on the computer of an Internet service provider. There the server is generally so configured that only connections from local computers (or a local network) are allowed. For administration you must therefore either create a telnet-/ssh connection or use a program that is executed locally on the server and is served via the internet (e.g., phpMyAdmin).

  • Name resolution of host names does not work correctly: In establishing a connection over a network, error 1130 arises (Host n.n.n.n' is not allowed to connect to this MySQL server). The most likely cause of this error is either the incorrect specification of the host name in the mysql.user table or an incorrect name-server configuration. Depending on the network configuration, the host names must be given in the column user.Host with or (more seldom) without domain name.

    A solution is to add domain names to the host names in the mysql.user table (uranus ® uranus.sol) or to remove this (uranus.sol ® uranus) and try again. (Do not forget FLUSH PRIVILEGES.)

    If that does not work, you can test with the commands hostname, host, and resolveip whether there are problems with name resolution. An emergency solution that almost always works (but is inflexible) is to give the IP number instead of the host name in the user.Host column. There are many other suggestions about dealing with host-name problems earlier in this chapter.

  • User name or password are incorrect: Watch out for typos! Note as well that not only user name and password must correspond, but the host name as well. (Thus a connection is usually possible only from certain specific computers.) Also, read the previous point relating to resolution of the host name; perhaps that is where the problem resides.

    Note that the user.Password column does not contain passwords in plain text, but in encrypted form. If you wish to change a password with SQL commands, you must use the function PASSWORD("xxx").

  • An incorrect entry was used in the mysql.user table: When user x attempts to register with computer y, the MySQL server compares the entries in the user table in a particular order: First, entries are considered whose Host character string is unique, and only then Host entries with wild cards (% and _). Within these two groups, again unique User strings are preferred to those with wild cards.

    The result of this order of precedence is that user abc on the local computer (localhost) will be unable under certain circumstances to register, although in the user table there is an entry Host='%' / User='abc'. The reason is that in the default setting of access privileges there is also an entry Host='localhost' / User=''. This entry is given precedence to the first one because there the host name is given explicitly.

    To solve the problem, either add a second entry Host='localhost' / User='abc' to the user table or delete the entry Host='localhost' / User=''. I would recommend the second variant, since that entry represents a security risk.

  • No user name was specified: If you do not specify a user name in your program for making a connection, then the login name of the account under which the program was launched is given automatically. In programs that are launched interactively, this is your login name.

    With programs that run over a web server (PHP or JSP scripts, Perl CGI files, etc.), the account name of the web server is used. For security reasons the web server usually runs not as root (Linux) or with administrator privileges (Windows), but in a separate account, such as wwwrun or apache. The problem is now that in the mysql access tables, the user wwwrun or apache is unknown. Therefore, access to the database is denied. Therefore, do not forget in script files to specify the user name for the connection to MySQL explicitly.

  • The connection succeeds, but access to the database is impossible: This error occurs immediately if you specify the desired database during the establishment of the connection. However, the error cannot occur until you select the desired database (USE dbname). Error message 1045 is, for example, Access denied for user ... to database ... .

    The most likely cause of the error is that the user in fact does not have access rights to the database. Perhaps in GRANT you have specified only the Usage privilege (which allows a login, but not the actual use of a database). Execute the command GRANT SELECT, INSERT ... ON dbname.* TO name@hostname to allow database access to dbname.

    If the MySQL server is running on the computer of an Internet service provider (ISP),then the server is generally so configured that only local access is possible. In other words, your PHP or Perl scripts run without problems (because they are executed on the same computer), but you cannot access your databases from home, say, with the MySQL Control Center.

    Here the issue is the correct (because secure) setting of the access privileges. You will find scarcely an ISP that allows MySQL connections from an arbitrary computer on the Intenet. You must therefore use programs for administration that run locally on the computer of the ISP (e.g., phpMyAdmin).

  • It is impossible to create a local TCP/IP connection: This problem usually occurs under Unix/Linux. A local connection succeeds only if with option -h no computer name or IP number is specified. The most likely cause is a problem with the resolution of the host name. As a rule, you must add the domain name in the column user.Host, and then it works. Another cause can be the local network configuration (file /etc/hosts). We have already given some tips in this chapter especially for Red Hat and SuSE Linux.

  • The local connection fails for Java programs: This problem is usually connected with the previous point, since Java programs, in contrast to most other MySQL clients, generally use TCP/IP (and not a socket file). Have a look in Chapter 15, where there are some trouble-shooting tips.

    Another cause of error can be the incorrect installation of Connector/J (see Chapter 15), but then an error occurs in the attempt to use JDBC (java.lang.ClassNotFoundException: com.mysql.jdbc.Driver).

  • Port 3306 is blocked: Between the MySQL server and your program there is a firewall that is blocking port 3306. This problem can occur only when your program and the MySQL server are running on different computers. If you manage the firewall yourself, you must clear port 3306; otherwise, you must ask the administrator to do so.

  • The MySQL server crashes at every attempt at a TCP/IP connection: This problem occurred in the past with Linux distributions. Since the server was immediately restarted, the problem was not always correctly identified. (The error message is usually Lost connection to MySQL server during query.)

    A solution to this problem is frequently an update of the glib library or the installation of the MySQL package from www.mysql.com. (The MySQL packages included in the distributions are not always optimally configured. This has been particularly bad in the case of Red Hat 8.0 and SuSE 8.1. The MySQL documentation therefore recommends using only versions of MySQL compiled by the MySQL team, and after a number of negative experiences, I agree.)

  • An update of the MySQL server causes problems: In MySQL 4.0 the security system was greatly expanded. If you are moving your database system from version 3.23.n to 4.n, you must be sure to bring the mysql tables into conformity with the new security system. For this, there is provided the script mysql_fix_privilege_tables (see also Chapter 10). This script creates new columns (vis-à-vis MySQL 3.23) in the mysql tables, but leaves the settings of these columns in the default setting N. You may need to take a close look at the new access privileges and explicitly grant certain individual new privileges, such as Lock, Create_tmp_table for normal users, and Super for administrators.

    Pointer

    An additional list of possible causes of error and some tips for alleviating them can be found in the MySQL documentation:

    http://www.mysql.com/doc/en/Access_denied.html

The Special Case of Named Pipes

Named pipes represent a mechanism by which two programs can communicate under Windows NT/2000/XP. Data exchange follows the first-in-first-out (FIFO) principle.

MySQL supports named pipes only under Windows NT/2000/XP, and not under Windows 9x/ME or under Unix/Linux. Named pipes can be used only when the server is appropriately compiled (mysqld-nt and mysqld-max, but not mysqld-opt) and when the configuration file Windows\my.ini contains the option enable-named-pipe in the group [mysqld]. By default, this is not the case, since in the past, open named pipes to the MySQL server caused problems in shutting down Windows.

If you wish to establish an explicit named-pipe connection, specify a period (.) as host name or use the option --pipe. Even if you specify no options, a named pipe will be used if the server permits it:

C:\> mysql -u name -p
C:\> mysql -h . -u name -p
C:\> mysql -h localhost --pipe -u name -p

If you wish to create a connection explicitly over TCP/IP, you must specify the host name:

C:\> mysql -h computername -u name -p
C:\> mysql -h localhost -u name -p

Whether named pipes are permitted can be determined in the program mysql with the command status.

Further Tips for Error-Checking

If you believe that your mysql database is properly configured, yet database access fails nonetheless, stop the MySQL server, add temporarily skip-grant-tables to the [mysql] section in my.cn or my.ini and restart the server. Now everyone has access to all data. If access now succeeds, you at least know with certainty that the problem resides in the mysql tables. Do not forget to remove skip-grant-tables from the configuration file.

MySQL manages temporary storage (cache) with a list of most recently used IP addresses and associated host names. This cache makes it possible for the usually time-consuming process of name resolution to proceed efficiently. However, if you change your network configuration without restarting the MySQL server, it can happen that this cache contains incorrect, that is, no longer valid, entries. To run the MySQL server without this cache, add skip-host-cache in the [mysql[ section of my.cnf or my.ini and restart the server.

Unfortunately, there is no way to make the MySQL server save precise information as to why a login attempt failed. In the error log (file hostname.err; see Chapter 10) is mentioned only the IP address of the login attempt, but not the course of name resolution, which entries in the mysql tables were looked at, etc.


System Security

Up to this point we have dealt in this chapter with issues of security as they apply directly to MySQL. In reality, of course, your data within a MySQL database are only as secure as the underlying operating system. (It is assumed that MySQL is running under an operating system that is equipped with security mechanisms. Security under Windows 9x/ME is a contradiction in terms, and it is to be hoped that no one will get the wild idea of running a public MySQL server under Windows 9x/ME.)

To secure your MySQL database system you should begin by asking the following questions:

  • Are there security loopholes that would place an attacker in a position to log in as root (Unix/Linux) or with administrator privileges (Windows NT/2000)?

  • Are the logging files secured? There, for example, the passwords can be found in plain text.

  • Are script files containing passwords in plain text adequately secured? (And if full-fledged security is impossible, is it ensured that the user name/password combination in these files is provided with minimal access privileges? Is it ensured that the given password can really be used only for this MySQL login, that for convenience the same password as for root was not used?)

An additional issue concerns the data within a database. You should prepare yourself for the worst-case scenario—the possibility that an attacker gains access to your data—by encrypting key information within the database:

  • If you store login information in the database (for example, the login information for users of your web site), then do not store the passwords in plain text, but only as MD5 check sums. These check sums are sufficient to verify a password. (The drawback is that a forgotten password cannot be restored from the check sum.)

  • If you store even more sensitive data (credit card information, say, to take a particularly relevant example), such data should be encrypted. You should ask whether MySQL is a suitable database system for managing such data. If you store such critical data, you should engage security experts to make your system as secure as possible. Anything less should be considered gross negligence.

Note that MySQL itself can present a security risk that can be used for access to the entire operating system. For example, a MySQL user with File privileges can read and alter files in the local file system. The extent of this ability depends on how the file system itself is secured (access privileges for directories and files) and under what account the MySQL server (and thus mysqld)isrunning.

In no case should the server be run as root under Unix/Linux or with administrator privileges under Windows, since then, every MySQL user with File privileges would have the run of the entire file system.

With all current Linux distributions MySQL is already correctly configured. You can check this with the following command:

root# ps au | grep mysqld
root
... /bin/sh /usr/bin/mysqld_safe ...
mysql ... /usr/sbin/mysqld ...
mysql ... /usr/sbin/mysqld ...
...

This means that the script mysqld_safe is executed by root, but the various threads of the MySQL server mysqld by the user mysql. (The script mysqld_safe must be executed by root. It serves only to start mysqld. It runs until the MySQL server is explicitly shut down. The script restarts mysqld automatically if the server crashes.)

If mysqld is executed as root,the following entry in /etc/my.cnf can help:

# file /etc/my.cnf
[mysqld]
user = mysql
Pointer

A nice example of what can happen when mysqld is executed as root is shown by the text located at

http://www.dataloss.net/papers/how.defaced.apache.org.txt

where it is described how the web server of apache.org was attacked in May 2000. The document is somewhat old now, but the principle described therein has not lost its validity.

Tip

If you set up MySQL as a database server only within a local network (that is, so that it is inaccessible from outside) and you use an IP packet filter as a firewall, then as an additional security measure you should close the IP port 3306. The result is that the MySQL server cannot be addressed directly from outside the network.

Internet access over dynamic web sites is still possible in spite of all security measures, since dynamic web sites are generated from the locally running web server (which in turn calls a script interpreter). The MySQL server is thus addressed locally, and external communication is accomplished not via port 3306, but via the HTTP protocol (normally port 80). If there is a security loophole there, then closing port 3306 will not be of any help.
Pointer

We may sum up by saying that one can achieve genuine security of MySQL only if one has a deep understanding of the operating system itself, and this cannot be discussed here, not least because I myself know too little about such issues, especially in regard to security mechanisms under Windows NT/2000.

However, there are many books on operating systems—completely independent of MySQL—that deal with the topics of administration and security.


Chapter 10: Administration and Server Configuration

Overview

This chapter describes the administration and configuration of the MySQL server. Among the most important themes that we treat are backups, logging files, migration of databases from one computer to another (or from one version of MySQL to a newer one), and import/export of text files. This chapter also goes specifically into the maintenance of MyISAM and InnoDB tables (such as the maintenance of InnoDB tablespace files).

Some advanced topics are treated in this chapter: Whether you change the character set of the MySQL server, set up a replication system, or wish to achieve greater processing speed, this chapter gives the relevant introductions and tips.

Finally, this chapter goes into the painful question of the various options for administration of databases that are present at an Internet service provider. (The problem is that in such a case you are not the database administrator and thus have very limited access rights.)


Basic Administration

Pointer

Some additional topics related to administration of MySQL are discussed in the following chapters:

  • Chapter 4: Use of user interfaces for administration (mysql, MySQL Control Center, phpMyAdmin, etc.)

  • Chapter 9: Access controls, mysql databases

  • Chapter 19: Reference to mysqld, mysql, mysqladmin, etc.

This section collects information on the elementary aspects of administration and refers the reader to other places in this book where additional details are to be found.

Basic Server Configuration

Setting the root Password

Afresh installation leaves MySQL completely unprotected. As soon as you begin to use MySQL for storing data, you should secure MySQL with a password. Under Unix/Linux the following commands suffice. (As computername the complete network name of the computer must be given.)

root# mysqladmin -u root -h localhost password xxx
root# mysqladmin -u root -h computername password xxx

The process is somewhat more complicated under Windows, since the default configuration is even more insecure, and mysqladmin frequently does not function as it should.

Pointer

The MySQL access system was described in detail in Chapter 9. In particular, in the section "First Aid" you will find a concrete introduction to securing MySQL under both Windows and Unix/Linux.

Personalizing the MySQL Server

If you have followed the installation instructions presented in Chapter 2, then the MySQL server should be running mysqld and should be functioning without any problems for most applications. But if you have particular wishes, say you need a particular sort order or wish to optimize performance, then you can specify a number of options for mysqld. Some important options are described in the course of this chapter. A reference section appears in Chapter 19.

Configuration File for Setting Options

Both at launch of mysqld and during the execution of administrative tools it is tiresome always having to specify the same old options. Therefore, one may specify these options in a configuration file:

Validity | Windows | Unix/Linux
global options (for both mysqld and administrative tools) | C:\my.cnf, Windows\my.ini | /etc/my.cnf
user-specific options (only for administrative tools) |  | ˜/.my.cnf

The server-specific options in the configuration file begin with the line [mysqld]. The following options have the effect that in sorting tables, the German alphabetic sort order is to be used.

# Example of the server-specific part of
# /etc/my.conf and Windows\my.ini
[mysqld]
default-character-set = latin1-de

Restarting the MySQL Server

Changes in configuration files become effective only after a restart of the affected program. This holds, of course, for the MySQL server as well. Thus, if you change server-specific options, you must restart the server (which is to be avoided as much as possible with a running system).

Under Windows you execute the restart most simply via WinMySQLadmin. (If you are working under NT/2000, you can also use the system administration services dialog.)

Under Unix/Linux you can use the appropriate Init-V script for restarting, which you typically execute first with the parameter stop and then with the parameter start.

Red Hat: root# /etc/init.d/mysql stop
         root# /etc/init.d/mysql start
SuSE:    root# /etc/init.d/mysql stop
         root# /etc/init.d/mysql start

Using Administrative Tools

With MySQL you get a number of administrative tools (such as mysqladmin and myisamchk), some of which are described in some detail in this chapter. A common feature of all these tools is that they are command-oriented programs. That is, there is no graphical user interface, and the tool is used with options and commands.

Under Unix/Linux the tools are executed in a shell window, while under Windows they are executed in a command window. (Chapter 4 contains tips on the optimal configuration of the command window as well as the proper setup of the Windows system variable PATH.)

Most of the administrative tools can be used with their full functionality only if you sign in as root and provide your password. To do this you specify the options -u and -p at launch of mysqld. For example, with the following command all active logging files are closed and reopened (with the result that in update protocols the sequential file ending is increased by 1).

root# mysqladmin -u root -p flush-logs
Enter password: ******
Remark

Some of the commands described in this chapter, generally those involving shell or Perl scripts, are available only under Unix and Linux, and not under Windows.

Creating a Database Structure (mysqlshow)

The command mysqlshow helps in obtaining a quick overview of the databases, tables, and columns managed by MySQL. If the command is executed without parameters, it simply returns a list of all databases managed by MySQL. If a database is specified, then mysqlshow shows a list of the tables contained within that database. If a table is specified in addition to the database, then mysqlshow displays a list of the columns of that table, including its properties (data type, default value, etc.).

In the last parameter the wild cards _ and % can be used to filter the indicated data. The same information obtained with mysqlshow can also be obtained with various variants of the SQL command SHOW. The advantage of mysqlshow is that the program is well suited for the automation of administrative tasks.

> mysqlshow -u root -p
Enter password: xxx
Databases:
  books
  myforum
  mylibrary
  mysql
  test
      . . .    >  mysqlshow -u root -p mysql
Enter password: xxx
Database: mysql
Tables:
  columns_priv
  db
  host
  tables_priv
  user
>  mysqlshow -u root -p books author
Enter password:  xxxxxx
Database: books Table: author Rows: 10

 Field      Type        Null  Key  Default  Extra           Privileges
 authorID   int(11)           PRI           auto_increment  select, . . .
 author     varchar(60)                                     select, . . .

Executing Administrative Commands (mysqladmin)

As the name of the program implies, mysqladmin assists in the execution of a variety of administrative tasks:

  • Create and delete databases

  • Change a user's password

  • Input again the privileges database mysql

  • Update database and logging files (clear buffer or intermediate storage)

  • Determine status information and variables of the MySQL server

  • List and store MySQL Processes

  • Test connection to MySQL server (ping)

Shut down the MySQL server (shutdown)

The following examples demonstrate a few possible applications:

> mysqladmin -u root -p create newDatabaseName
Enter password: xxx
> mysqladmin -i 5 ping
mysqld is alive
mysqld is alive
...
> mysqladmin status
Uptime: 435152  Threads: 2       Questions: 26464  Slow queries: 3
Opens: 140      Flush tables: 1  Open tables: 0    Queries per second avg: 0.061
> mysqladmin -u root -p processlist
Enter password: xxx
Id    User  Host       db      Command  Time  State  Info
1     ODBC  localhost  books3  Sleep    7
196   root  localhost          Query    0                 show processlist
...

Each time mysqladmin is called, only one operation can be executed. Most mysqladmin operations can be carried out as well with SQL commands (for example, with mysql or with a client program). The advantage of mysqladmin is that the program is easily adapted to the automation of administrative tasks.

Pointer

Additional examples of the use of mysqladmin in connection with the setting of privileges, access, and passwords can be found in Chapter 9.

Under Windows, mysqladmin frequently causes problems in setting up passwords, because it does not correctly interpret the computer name localhost. In such cases, you should change the password with SET PASSWORD, as described in Chapter 9.

A reference to all mysqladmin options and commands can be found in Chapter 19.


Backups

Backups are doubtless one of the most important tasks of a database administrator. MySQL offers a variety of methods for executing a backup.

  • The classical way, so to speak, uses the command mysqldump. This command returns a file with SQL commands for generating the tables of the database and filling them anew with data. Backups using mysqldump are comparatively slow, but they offer maximal portability. Therefore, mysqldump is used for database migration (for example, to copy a database from one MySQL installation to another).

    The command mysql is used to recreate the database.

  • Significantly faster is simply to copy the database directories at the system level. This is quite efficient and secure if the MySQL server is stopped for this purpose. However, if the server is to continue running, then it must be ensured that during the copying process no changes are allowed to be made in the database. This task is taken care of by the script mysqlhotcopy.

    To recreate the database it is necessary merely to copy the backed-up directory into the MySQL database directory.

    Note that this form of backup is possible only for MyISAM tables (and not for InnoDB tables).

  • One can use logging files in combination with regular backup files for incremental backups. However, that is practicable only as long as the number of changes in the database is not too large. Backup strategies based on logging files are not handled in this section, but in the section on logging.

    Another variant of securing data is replication. This mechanism allows a database to be synchronized on two different computers. For databases that are changed frequently this can result in a high demand on communication between the computers. If you really want nothing more than to keep two copies of your database on two different hard drives, then it is more efficient to mirror a hard drive using RAID. Replication is not dealt with in this section, but in the section of this chapter on replication.

HotBackups:Both the execution of mysqldump and the copying of database directories can take place while the database is in operation. However, the integrity of the backup is ensured only if read lock is executed during the backup for the affected tables.

The result is, on the one hand, that the backup can be executed only when the table is not blocked by a write lock. On the other hand, no client can access the tables during the execution of the backup. Since the backup of large databases can take a relatively long time (especially if you use mysqldump), blocking is quite burdensome.

To circumvent this problem, commercial database systems offer hot backup mechanisms that permit a backup while the system is running without the complete blocking of entire tables. For MyISAM tables, MySQL is unfortunately incapable of this (despite the promises of mysqlhotcopy). If you use InnoDB tables, however, you can execute hot backups with the supplementary program InnoDB Hot Backup (see http://www.innodb.com/hotbackup.html).

Securing Databases (mysqldump)

The command mysqldump returns a long list of SQL commands required for the exact recreation of a database. Normally, the program returns a CREATE TABLE command for each table in order to generate the table together with indexes, as well as numerous INSERT commands (one for each record). The following example shows the result of mysqldump for the authors table of the mylibrary database presented in Chapter 5. (Lines that begin with -- are considered comments in SQL.)

> mysqldump -u root -p mylibrary author
Enter password: xxx
-- MySQL dump 9.07
-- Host: localhost Database: mylibrary
-- Server version
4.0.12-nt-log
-- Table structure for table 'authors'
CREATE TABLE authors (
  authID int(11) NOT NULL auto_increment,
  authName varchar(60) NOT NULL default '',
  PRIMARY KEY (authID),
  KEY authName (authName),
  FULLTEXT KEY authName_2 (authName)
) TYPE=MyISAM;
-
-- Dumping data for table 'authors'
INSERT INTO authors VALUES (1,'Kofler Michael');
INSERT INTO authors VALUES (2,'Kramer David');
INSERT INTO authors VALUES (3,'Orfali Robert');
INSERT INTO authors VALUES (4,'Harkey Dan');

Provided that the option --tab is not used, mysqldump displays the list of commands on the monitor. Usually, however, it makes more sense to direct the output to a file using > filename.

Caution

When output is redirected to a file under Windows, one has the problem that the line Enter password will be entered into the target file. When the file is later read, a syntax error will occur, since Enter password is not an SQL command. This is not a pretty error. However, the data can be input without difficulties. If you wish to avoid the error message, you must either delete the line with a text editor after the file has been dumped, or else specify the password directly with the option --password=xxx,which is insecure.

Under Unix/Linux this error does not occur, because a distinction is made between the channel for standard output and that for error messages. The line Enter password is normally displayed via the error channel on the monitor, while the SQL commands are stored by the output channel in the file.

Backup of All Databases

The following command creates a backup of all databases managed by MySQL (provided that the user has root access) and stores these in a single file. During the backup, the database is blocked for all write operations:

> mysqldump -u root --password=xxx --opt --all-databases > backup.sql

Syntax of mysqldump

The general syntax of mysqldump is the following:

mysqldump [options] dbname [tables] > backup.sql

If no tables are specified, then mysqldump writes all the tables of the database dbname into the file backup.sql. Optionally, the backup can be limited to specific tables. If you wish to back up more than one database or even all of them, then the following syntax variants are what you use:

mysqldump [options] --databases dbname1 dbname2 ...
mysqldump [options] --all-databases

The details of the backup can be controlled by a large number of options. An extensive description and reference to all the options appears in Chapter 19. To execute a simple backup, a single option usually suffices, --opt, which has the following effect:

  • During the backup read lock is executed for all tables.

  • The resulting backup file is as small as possible (one INSERT command for all records of the table).

  • The resulting file contains DROP TABLE commands to delete existing tables when the database is restored.

  • All features of the database are retained (including MySQL-specific features that could interfere with migration to a different database system).

On the other hand, --opt is not to be recommended if you use InnoDB tables or if the resulting files are to be backward compatible (such as moving a database from a MySQL server 4.0.n to an older server 3.23.n).

Remark

Perhaps you are aware that there is also a command mysqldumpslow. This command has nothing to do with mysqldump, buthelps in the evaluation of a special logging file in which all SQL commands that took a particularly long time to execute are recorded.

mybackup

The Perl script mybackup is currently not included with MySQL. However, you can find it on the Internet: http://www.mswanson.com/mybackup/

The script mybackup uses mysqldump. The advantage over mysqldump is that the backup files (one per database) are written into a specified directory and there immediately compressed with gzip.

Restoring a Database (mysql)

There is no direct inverse operator for mysqldump, simply because the old workhorse mysql is completely satisfactory for this purpose.

Remark

Tables are automatically recreated in the type that they previously possessed. Thus mysqldump and mysql are well suited as tools for securing data independent of table type (MyISAM, BDB, etc.).

The re-creation of a database works even if you use a different (newer) version of MySQL or if MySQL is running on a different computer (under a different operating system).

Restoring a Single Database

If backup.sql contains only a single database, then the restore command looks like this:

> mysql -u root -p databasename < backup.sql
Enter password: xxx

The database databasename must already exist. If that is not the case, then you can easily create it with mysqladmin create databasename.

Restoring Several Databases

If several databases were backed up with mysqldump, then backup.sql contains the requisite CREATE DATABASE commands for creating the databases anew if they do not yet exist. There is no necessity to create the databases first. You also do not need to specify a database when mysql is called.

root# mysql -u root -p < backup.sql
Enter password: xxx

Reading In Databases and Tables Interactively

You can read in *.sql files interactively in the program mysql, that is, without restarting this program each time. For this, you use the command SOURCE or its abbreviated form /. and specify the file name of the *.sql file. (Under Windows, directories can be separated with either / or \. The file name does not need to be placed in quotation marks.)

Now mysql executes every SQL command in the file. However, the many outputs to the monitor (Query OK, 1 row affected) that are normally displayed are burdensome. They can be prevented by launching mysql with the option --silent:

root# mysql -u root -p --silent
Enter password: xxx
mysql> USE dbname
mysql> \. /tmp/backup.sql

Speeding Up the Reading of Databases and Tables

With small data sets (up to a few megabytes), the reading in of data is so fast that you do not need to worry about speed optimization. However, if you are dealing with large data sets, then a few thoughts in the direction of optimization are in order:

  • In executing backups with mysqldump you should use at least the following options:

    --add-drop-table (existing tables are deleted)

    --all (complete MySQL syntax)

    --extended-inserts (compact INSERT commands)

    --disable-keys (more efficient index management)

  • In the case of InnoDB tables, you should consider some of the tips for optimization presented later in this chapter.

Restoring InnoDB Tables (Integrity Rules)

In reading in InnoDB tables for which there exist integrity rules (foreign key constraints), you must execute SET foreign_key_checks=0 and then SET foreign_ key_checks=1 before reading in the tables. The reason is that a condition for these rules always being satisfied is that the records be inserted in the correct order. This is difficult, and often impossible, to ensure when data are reread in. Therefore, turning off these rules is a necessity if the tables are to be reread in without error.

Fast Backups (mysqlhotcopy)

For a considerable time, the Perl script mysqlhotcopy has been provided with the Unix/Linux version of MySQL. This script is supposed to help in improving the speed of backups executed with mysqldump. The basic idea of the program is first to execute a read lock for the specified databases and then with FLUSH TABLES to ensure that the database files are actually in their current versions. Then the database files are directly copied.

Caution

mysqlhotcopy creates a direct copy of table files. Therefore, the command can be used only with MyISAM tables.

If you use mysqlhotcopy on InnoDB tables, only the *.frm files will be copied, and there will be no error message.
Tip

The documentation to mysqlhotcopy is obtained with the following command. Instead of /usr/bin you may have to give another path:

user$ perldoc /usr/bin/mysqlhotcopy

Making a Backup

In the simplest case, the use of mysqlhotcopy is as follows:

root# mysqlhotcopy dbname1 dbname2 dbname3 backup/

With this the databases dbname1, dbname2, dbname3 are copied into the specified backup directory. (This backup directory must already exist.) Each database is copied into a subdirectory named for the database.

Options

The following list gives merely the most important options. A full description of all the options can be found in the documentation:

  • --allowold overwrites existing backup files.

  • --keepold archives existing backup files into the directory dbname_old. If this directory already exists, then its contents are overwritten.

  • --flushlog has the effect that MySQL changes made in the databases after the backup are written to a new logging file. This option makes sense only if the logging files are to be used for an incremental backup.

  • --noindices copies for (My)ISAM tables only the actual data, not the index files. (More precisely, the first two kilobytes of the index files are copied. This makes possible an uncomplicated, though slow, restoration of the index files with myisamchk -r. But the backup can be executed more quickly, because fewer data must be copied.)

The options -u, -p, -P, and -S, as well as --user=, --password=, --port=, and --socket=, function as they do with all MySQL tools, but with one exception. In the case of -p, the password must be given here (while all other tools provide interactive password input, which is more secure).

Restoring a Database

To restore a database you simply copy the relevant database directory into MySQL's data directory. If you execute this operation as root, you must then specify the owner of the database files with chown. (The owner must have the same account under which mysqld is running, usually mysql.)

root# chown -R mysql.mysql /var/lib/mysql/dbname
Caution

If the database to be restored already exists, the files involved in an operational MySQL cannot simply be overwritten. First execute DROP DATABASE.

It is possible to recreate data only if the version of MySQL that is running is compatible with the version under which the backup was executed. This is the case with the various MySQL versions 4.n. However, it is not guaranteed that a backup executed with MySQL 4.0 will be able to be restored with MySQL 4.n. (With each change in the main version number the format of database files can conceivably change.)


Database Migration

The term "migration" applied to databases denotes the transport of a database from one system to another. There are many reasons that can account for the migratory instinct appearing in a database:

  • installation of a new database server

  • transfer of a development system (for example, on a local computer) to a production system (on the ISP's computer)

  • aMySQL update (for example, from version 3.23 to 4.0)

  • a change in database system (for example, from Microsoft SQL Server to MySQL)

Transfer of Databases Between MySQL Systems

Migration between MySQL systems is generally carried out with the backup tools mysqldump and mysql, which we have previously described.

If the tables are in MyISAM format and compatible versions of MySQL are running on both computers (say, 3.23n), then the migration can be effected by simply copying the database files. This holds beginning with version 3.23 even if MySQL is running under different operating systems.

The main advantage of direct copying of MyISAM tables as opposed to the use of mysqldump/mysql is, of course, the much greater speed. Note, however, that you must recreate all the indexes with myisamchk if MySQL on the other computer uses a different character set (that is, a different sort order).

MySQL guarantees compatibility of database files only within main versions (such as from 4.0.17 to 4.0.18), and not between major updates (such as from 3.22 to 3.23 or 3.23 to 4.0). In fact, even major updates generally cause few problems (see the next section).

Tip

If you wish to change only the format of database files, say from MyISAM to InnoDB, then you do not need to go through the migration process. The command ALTER TABLE tblname TYPE=newtype will be adequate to meet your needs. If you wish to convert a large number of tables, then under Unix/Linux the Perl script mysql_convert_table_format will save you a great deal of effort. (The script assumes that a connection to the MySQL server can be established.)
Tip

If you carry out the migration with mysqldump/mysql, then you do not necessarily have to create (possibly enormous) files. You can pass the output of mysqldump directly to mysql.

The following command demonstrates the usual way of proceeding. It assumes that the command is executed on the computer with the source database and that the data are transported to a second computer (hostname destinationhost). There the database in question must already exist, and the access privileges must allow access to the source computer.

For space considerations the command is broken over two lines:

root# mysqldump -u root --password=xxx --opt sourcedb | \
    mysql -u root --password=yyy -h destinationhost dbname
Pointer

A special case of migration, namely, setting up a database on an ISP, is described toward the end of this chapter. The usual problem in doing this is the possession of inadequate access privileges to the ISP computer.

Pointer

If you copy databases from Windows to Unix/Linux, there can arise problems with the case-sensitivity of file extensions. The Perl script mysql_fix_extensions solves this problem (see Chapter 19).

MySQL Update from Version 3.23 to Version 4.0

Usually, an update to the MySQL server (such as from 4.0.7 to 4.0.8) is done in such a way that the server is stopped, the old version deinstalled, and the new version installed.

For the database mysql, before deinstallation a backup with access privileges must be carried out, since this database will be deleted and overwritten. (Your best course is to rename the mysql database directory mysqlold before the deinstallation.) Like the configuration file for the server, all other database files will be carried over unchanged.

Tip

Although an update usually causes no problems, it is highly recommended before any server update to create a complete backup with mysqldump of all databases. Better safe than sorry!

This simple process is possible in updating from 3.23n to 4.0.n. However, there are also some details to keep track of:

  • Configuration file: Some of the configuration options for the MySQL server have new names or were deleted. Make sure that no such options appear in Windows\my.ini or /etc/my.cnf. Otherwise, the server cannot be started. The following options have had their names changed:

    Old name

    New name

    enable-locking

    external-locking

    myisam_bulk_insert_tree_size

    bulk_insert_buffer_size

    query_cache_startup_type

    query_cache_type

    record_buffer

    read_buffer_size

    record_rnd_buffer

    read_rnd_buffer_size

    skip-locking

    skip-external-locking

    sort_buffer

    sort_buffer_size

    warnings

    log-warnings

  • mysql database: In MySQL 4.0, some tables of the mysql database were expanded by additional columns. (The mysql database serves, as described extensively in Chapter 9, for managing access privileges under MySQL.)

    Under Unix/Linux you can simply update the database mysql restored from an earlier version using the script mysql_fix_privilege_tables. To execute the script, you must give the root password (unless the password is empty). Then you may have to set certain privileges by hand, so that existing programs continue to be executed correctly. (This affects particularly the new privileges Create Temporary Table and Lock Tables, which by default are usually not specified, but which are required by many programs.)

    Under Windows, the further use of the mysql database is more complicated: Although the script mysql_fix_privilege_tables is included (directory scripts), its execution is impossible, due to the absence of a suitable shell interpreter. The best approach is to restore the old mysql database under another name (e.g., mysqlold) and then copy the entries into the new mysql tables using INSERT INTO ... SELECT. The following commands show how to proceed for the user table:

    USE mysql
DELETE FROM user
INSERT INTO user (Host, User, Password,
    Select_priv, Insert_priv, Update_priv, Delete_priv,
    Create_priv, Drop_priv, Reload_priv, Shutdown_priv,
    Process_priv, File_priv, Grant_priv, References_priv,
    Index_priv, Alter_priv)
SELECT * FROM mysqlold.user

    You proceed analogously for the other mysql tables. With these commands, only the old privileges, of course, are copied. The new ones must be granted manually (this is not the case by default). In particular, root should have all privileges! Finally, you must execute FLUSH PRIVILEGES for the changes to take effect.

  • If you have been working with ISAM tables, then a transfer to tables of type MyISAM is to be recommended. For conversion, execute ALTER TABLE tblname TYPE=MYISAM or use the Perl script mysql_convert_table_format (only under Unix/Linux). (The script assumes that a connection to the database can be established.)

    The ISAM table format (the predecessor to MyISAM) is still supported in MySQL 4.0, but it is considered obsolete (deprecated). By version 5.0 at the latest, ISAM tables will finally cease to be able to be used.

    Pointer

    Alistoffurther details that distinguish MySQL 3.23 from 4.0 can be found in the MySQL documentation, in the section "Upgrading/Downgrading":

    http://www.mysql.com/doc/en/Upgrade.html

Updating MySQL from Version 4.0 to Version 4.1

In principle, updating is accomplished by deinstalling the old version of the server and installing the new one. Then you must change only the table mysql.user. Version 4.1 provides a new password column in this table, which is supposed to improve security. Under Unix/Linux you simply execute the script mysql_fix_privilege_tables to introduce this column. It is unclear how best to proceed under Windows.

Remark

As this book was being completed, there was still no stable version of MySQL 4.1. Therefore, it could be that the update process will require additional modifications to databases or configuration files.

Changing the Database System

For migrating to MySQL from another database system or vice versa there is no universal solution. Almost every database system offers a tool, comparable to mysqldump, that represents the contents of the database as an SQL file. The problem is that the resulting files are seldom precisely compatible (for example, due to different column types or lack of ANSI SQL/92 conformity). With Find and Replace you can solve some of these problems. Under Unix/Linux the tools awk and sed can be helpful.

You will find converters for migration to MySQL for some database systems at the following addresses:

http://www.mysql.com/portal/software/index.html
http://www.mysql.com/downloads/contrib.html

If you are working under Windows, then ODBC is often helpful. For example, you can use Access to import a database of an ODBC-compatible system and then export it to MySQL. Do not expect, however, that all the details of the definition of the columns will remain intact.

Pointer

Information on the use of Connector/ODBC, the ODBC driver for MySQL, can be found in Chapter 17. There you will also find a description of how to move databases between MySQL and Access and how to convert databases from the Microsoft SQL server to MySQL.


Importing and Exporting Text Files

Sometimes, the contents of a table should be written as efficiently as possible into a text file or read from such a file. MySQL offers several ways of doing this:

  • The SQL command LOAD DATA reads in a text file and transfers the contents into a table.

  • With mysqlimport there is a command available that is equivalent to LOAD DATA.Itis especially well suited for automating the importation of a script file.

  • The SQL command SELECT ... INTO OUTFILE writes the result of a query into a text file.

  • If you wish to automate exportation with a script, then the command mysqldump is to be recommended. Its functionality is similar to that of SELECT ... INTO OUTFILE.

  • In many cases you can use the universal tool mysql for implementing text, HTML, or XML exportation.

If none of the above commands suits your needs, then you will have to write your own script to assist you in importing or exporting. The programming language Perl was made for such tasks.

Special Characters in the Imported or Exported File

A common feature of LOAD DATA, SELECT ... INTO OUTFILE, mysqlimport, and mysqldump is the set of options for handling special characters in a text file. There are four options for this purpose, which as SQL commands look like this:

FIELDS TERMINATED BY 'fieldtermstring'
       ENCLOSED BY 'enclosechar'
       ESCAPED BY 'escchar'
LINES TERMINATED BY 'linetermstring'

  • fieldtermstring specifies the character string that separates the columns within the row (for example, a tab character).

  • enclosechar specifies a character that is permitted to appear in the text file before and after individual entries (usually a single or double quote character for character strings). If an entry begins with this character, then that character is deleted at the beginning and end. (The end of a column is recognized by fieldtermstring.)

  • escchar specifies the escape character that is to be used to indicate special characters (the default is the backslash). This is necessary if special characters appear in character strings of the text file that are also used for separating rows or columns. Moreover, MySQL expects code 0 in the form \0 (where the backslash is to be replaced by escchar).

  • linetermstring specifies the character string with which a row is terminated. With DOS/Windows text files the character string '\r\n' must be used.

Working with Character Strings, Numbers, Dates, and BLOBs

For all the commands introduced in this section there is a data format that must be followed exactly. In particular, for importation you must hold to the format expected by MySQL. For exportation you have somewhat more leeway, in that you can use SQL functions for formatting data in the SELECT command (such as DATE_FORMAT for formatting dates and times).

Moreover, there are four options that you can use to determine how rows and columns should be separated and how character strings and special characters should be indicated. (Details can be found in Chapter 18 under LOAD DATA and SELECT ... INTO OUTFILE.)

  • Numbers: For very large and very small numbers in the FLOAT and SINGLE formats one has the use of scientific notation in the form -2.3e-037.

  • Character Strings: Strings are not changed in importation and exportation (ANSI format, one byte per character). Special characters contained in the character string are marked by default with the backslash in order to distinguish these from the characters used for separation (e.g., tab, carriage return, linefeed).

  • BLOBs: Binary objects are treated byte for byte like character strings. Neither in importing nor exporting is there the possibility of using hexadecimal character strings (0x123412341234 ... ).

  • Date and Time: Dates are treated as character strings of the form 2003-12-31, and times as character strings of the form 23:59:59. Timestamp values are considered integers of the form 20031231235959.

  • NULL: The treatment of NULL is problematic. The following text assumes that the backslash is used as the escape character for special characters and the double quote character for indicating a character string. If you use other characters (options FIELDS ESCAPED BY '?' ENCLOSED BY '?'), then you will have to reconfigure the following paragraphs.

    In exporting with escape characters, NULL is represented by \N. In exporting without escape characters NULL is simply represented by the four characters NULL. However, NULL or \N is placed between double quote characters (though not if they are in a text or BLOB field) and can therefore be distinguished from character strings.

    In importing with escape characters, MySQL accepts NULL, \N, and "\N" as NULL. However, "NULL" is interpreted as a character string (consisting of the four characters NULL).

    Caution

    If you use neither escape characters for special characters nor "enclose" characters for character strings in importing and exporting, then the condition NULL cannot be distinguished from the character string NULL.

    In the character string \N the N must be uppercase. Note that \n is the newline character.

Importing with LOAD DATA INFILE

The syntax of LOAD DATA is as follows:

LOAD DATA [loadoptions] INFILE 'filename.txt' [duplicateoptions]
  INTO TABLE tablename [importoptions] [IGNORE ignorenr LINES]
  [(columnlist)]

The result is that the file filename.txt is imported into the table tablename. There are various options (see the SQL reference in Chapter 18) that can be given, as well as the column names for the table. To execute the command you need the File privilege (so that files can be read).

Example 1

The table importtable1 consists of five columns: one AUTO_INCREMENT column (id) and the columns a_double, a_datetime, a_time, and a_text.

The column name refers to its data type. The Windows text file import1.txt is to be imported into this table. (Here ® represents a tab character):

12.3 | ® | 12/31/1999 | ® | 17:30 | ® | text
-0.33e-2 | ® | 2000/12/31 | ® | 11:20 | ® | "text in quotes"
1,23 | ® | 31.12.2001 | ® | 0:13 | ® | "german text with äöü"

The import command looks like this:

USE exceptions
LOAD DATA INFILE 'c:\import1.txt'
INTO TABLE importtable1
FIELDS TERMINATED BY '\t'
      ENCLOSED BY '\"'
LINES TERMINATED BY '\r\n'
(a_number, a_date, a_time, a_string)
  Query OK, 3 rows affected (0.00 sec)
  Records: 3 Deleted: 0 Skipped: 0 Warnings: 2

A SELECT command demonstrates that the importation was only partially successful. In both the first and third lines, the date has been incorrectly interpreted. Moreover, in the third line the decimal number with the German comma for a decimal point has caused problems; namely, the part to the right of the decimal point has gone missing. In sum, take care to obey the MySQL formatting rules to the letter when preparing a file for importation:

SELECT * FROM importtable1
 id  a_number  a_datetime            a_time    a_string
------------------------------------------------------------------
 1       12.3  0000-00-00  00:00:00  17:30:00  text
 2    -0.0033  2000-12-31  00:00:00  11:20:00  text in quotes
 3          1  2031-12-20  01:00:00  00:13:00  german text with äöü

Example 2 (BLOB, NULL)

The starting point for our second example is the table importtable2 with columns id and a_blob. The second column is allowed to contain NULL.

  • The following Unix text file is to be imported into this table (again,® represents a tab character):

    1    ® NULL
2  ® "NULL"
3  ® \N
4  ® "\N"
5  ® \n
6  ® "\n"
7  ® 0x414243
8  ® "0x414243"
9  ® blob blob
10  ® "blob blob"

The text file is imported with the following command. (By default, the backslash serves as escape character, and the tab sign as column separator.) REPLACE means that existing data records are to be replaced.

USE exceptions
TRUNCATE importtable2
LOAD DATA INFILE '/tmp/import2.txt' REPLACE
INTO TABLE importtable2
FIELDS ENCLOSED BY '\"'
Query OK, 12 rows affected (0.00 sec)
Records: 10 Deleted: 2 Skipped: 0 Warnings: 0

In order to analyze the result of the importation in more detail, we shall use SELECT * and add the additional columns LENGTH(a_blob) and ISNULL. This enables us to distinguish the state NULL (length NULL) from the character string NULL (length 4):

SELECT *, LENGTH(a_blob) FROM importtable2
 id a_blob     LENGTH(a_blob) ISNULL(a_blob)
----------------------------------------
 1  NULL       NULL           1
 2  NULL       4              0
 3  NULL       NULL           1
 4  NULL       NULL           1
 5             1              0
 6             1              0
 7  0x414243   8              0
 8  0x414243   8              0
 9  blob blob  9              0
 10 blob blob  9              0

We see that in the first six records, NULL was treated correctly in only the first, third, and fourth records. In the second record, the character string "NULL" was stored, while in the fifth and sixth records a newline character was stored.

The attempt to read in a hexadecimal number as a binary object fails. Both times, 0x414243 is interpreted as the character string "0x414243" (and not, as intended in record 7, as hex code for the character string "ABC").

CSV Import

Sometimes, one wishes to import data into MySQL from a spreadsheet program like Excel. Such programs generally offer the possibility to store tables in CSV (comma-separated values) format. In principle, the importation of such files proceeds effortlessly. For files that were created with Excel under Windows, suitable import options look like this:

FIELDS TERMINATED BY ',' ENCLOSED BY '\"'
LINES TERMINATED BY '\r\n'

In practice, importation usually runs into trouble with the formatting of dates (usually in the form 12/31/2003, about which MySQL is clueless). In particular, with Microsoft software, the automatic country-specific formatting of numbers can cause problems when, for example, Excel suddenly represents a decimal number with a comma instead of a period for the decimal point. In such cases one can get help from a special import script for MySQL (or else you can program your own export filters for Excel).

Error Search

As the previous examples have shown, importing text is often problematic. Unfortunately, LOAD DATA returns in MySQL 4.0 merely a brief status output, which gives, among other things, the number of warnings:

Query OK, 15 rows affected (0.00 sec)
Records: 10 Deleted: 5 Skipped: 0 Warnings: 2

This result means that ten records were read in. Five existing records were replaced by new records (thus 15 rows affected). There were problems with two records (rows) detected by LOAD DATA (2 warnings). The nature of the problems, and which rows of the file were affected, remains a mystery.

Paul DuBois, author of several books on MySQL, has created a Perl script for such cases that copies text files line by line into a temporary file and reads them in with LOAD DATA. This is slow, but it provides the opportunity to identify defective rows exactly and to obtain information about possible sources of error. The script is described in MySQL Cookbook, Chapter 10, and at the Internet address http://www.kitebird.com/mysql-cookbook/examples-ld.php.

Starting with MySQL 4.1, error searching should be easier: With the commands SHOW WARNINGS and SHOW ERRORS all problems that arose with the execution of the last command are listed.

Importing with mysqlimport

If you wish to execute LOAD DATA not as an SQL command but through an external program, then MySQL offers the tool mysqlimport.(It relies on LOAD DATA.)

mysqlimport databasename tablename.txt

The command reads the specified file into a like-named table of the database databasename.(The table name is thus taken from the file name, with the file identifier being eliminated. Thus mysqlimport db1 authors.txt imports the data into the table authors of the database db1.)

Please note that the file normally is read from the file system of the MySQL server. If you are working at another computer and wish to read a file that is located there, you must use the option --local. The options --fields-terminated-by, --fields-enclosed-by, --fields-escaped-by, and --lines-terminated-by correspond to the SQL options described at the beginning of this section. These options should be placed in quotation marks, for example, "--fields-enclosed-by=+".

To carry out the importation demonstrated in the previous section (example 2) with mysqlimport, the file to be imported must first be renamed to correspond with the table name in importtable2.txt.Then the following command is necessary:

root# mysqlimport --local "--fields-enclosed-by=\"" \ exceptions /tmp/importtable2.txt

Under Windows (that is, for importing a Windows text file), the command would look like this (broken into two lines here for reasons of space):

> mysqlimport --local "--fields-enclosed-by=\"" "--lines-terminated-by=\r\n"
   exceptions c:\importtable2.txt

Exporting with SELECT ... INTO OUTFILE

With the command SELECT ... INTO OUTFILE we are dealing with a garden-variety SELECT, where before the FROM part, INTO OUTFILE is used to specify a file name and several possible options:

SELECT [selectoptions] columnlist
INTO OUTFILE 'filename.txt' exportoptions
FROM ... WHERE ... GROUP BY ... HAVING ... ORDER BY ... LIMIT ...

The result of the query is thus stored in the file filename.txt. With exportoptions various options can be specified for dealing with special characters (see the SQL reference in Chapter 18).

Example

The following example assumes that there exists a table exporttable in the current database that has the following contents (note in particular the field a_char in the second data record):

SELECT * FROM exporttable
 id  a_char     a_text     a_blob     a_date      a_time   . . .
-----------------------------------------------------------------
 1   char char  text text  blob blob  2001-12-31  12:30:00
 2   ' " \ ; +  adsf       NULL       2000-11-17  16:54:54

 . . .   a_timestamp      a_float    a_decimal  a_enum  a_set
---------------------------------------------------------------
         20001117164643   3.14159    0.012      b       e,g
         20001117165454   -2.3e-037  12.345     b       f,g

The data types of the columns are taken from the column names. The following lines show the result of an OUTFILE exportation without special options. The tab character is again indicated by a hooked arrow. The resulting file has two lines, which for reasons of space are here distributed over five lines:

SELECT * INTO OUTFILE '/tmp/testfile.txt'
FROM exporttable
1 ® char char  ® text text
  ® blob blob  ® 2001-12-31  ® 12:30:00  ® 20001117164643  ® 3.14159  ® 0.012
    ® b  ® e,g
2  ® '"\\;+  ® adsf  ® \N  ® 2000-11-17  ® 16:54:54  ® 20001117165454
  ® -2.3e-037  ® 12.345  ® b  ® f,g

In the second attempt a semicolon is used as column separator. All fields are enclosed at beginning and end with a double quote character. As escape character the backslash is used (the default setting). Of particular interest is the behavior in the field a_char:

'"\;+

becomes

"' \" \\ ; +"

The semicolon within the character string remains untouched, since it is clear on account of the double quote character where a_char ends. However, the double quote becomes a backslash to ensure that the character is not misinterpreted at the end of the character string.

Please note also that NULL is represented as \N, and in fact, without being enclosed in double quotes.

SELECT * INTO OUTFILE '/tmp/testfile.txt'
FIELDS TERMINATED BY ';' ENCLOSED BY '\"'
FROM exporttable
1;"char char";"text text";"blob blob";"2001-12-31";"12:30:00";
20001117164643;3.14159;0.012;"b";"e,g"
2;"' \" \\ ; +"; "adsf";\N;"2000-11-17";"16:54:54";
20001117165454;-2.3e-037;12.345;"b";"f,g"

In our third attempt, a semicolon is again used as a column separator. The plus sign (+) is used as optional field identifier (that is, only with character strings, dates, times, and BLOBs, and not with numbers). Finally, the exclamation mark (!) is used as escape character. Again the most interesting feature is the transformation of a_char:

'"\;+

becomes

+'"\;!++
SELECT * INTO OUTFILE '/tmp/testfile.txt'
FIELDS TERMINATED BY ':' OPTIONALLY ENCLOSED BY'+' ESCAPED BY '!'
FROM exporttable

Now NULL becomes !N (that is, for NULL, too, the changed escape character is used).

1;+char char+;+text text+;+blob blob+;+2001-12-31+;+12:30:00+; 20001117164643;
3.14159;0.012;+b+;+e,g+
2;+'"\; !++;+adsf+;!N;+2000-11-17+;+16:54:54+; 20001117165454;
-2.3e-037;12.345;+b+;+f,g+

Basically, the data stored in a text file with LOAD DATA can be read again into a table unchanged if the same options are used as with SELECT ... INTO OUTFILE.

Exporting with mysqldump

As an alternative to SELECT ... INTO OUTFILE there is the auxiliary program mysqldump. This program is actually primarily a backup program (see also earlier in this chapter and the references in Chapter 18).

The program mysqldump always stores entire tables (and not the result of a particular SELECT query). A further difference is that mysqldump normally does not return a text file with the raw data, but entire INSERT commands. The resulting file can then later be read in with mysql. To use mysqldump for text exportation you must specify the option --tab:

mysqldump --tab=verz [options] databasename tablename

With --tab a directory is specified. In this directory mysqldump stores two files for each table: tablename.txt and tablename.sql. The *.txt file contains the same data as after SELECT ... INTO OUTFILE. The *.sql file contains a CREATE TABLE command, which allows the table to be re-created.

As with mysqlimport, the representation of special characters can be controlled with four options: --fields-terminated-by, --fields-enclosed-by, --fields-escaped-by, and --lines-terminated-by. These options are analogous to the SQL options described at the beginning of this section. They should be set in quotation marks (for example, "--fields-enclosed-by=+"):

C:\> mysqldump -u root -p --tab=c:\tmp "--fields-enclosed-by=\"" exceptions exporttable
Enter password: ******

XML Exporting with mysqldump

If you execute mysqldump with the option --xml, you obtain as result an XML file (where, however, characters outside the 7-bit ASCII character set are not represented by Unicode, but by the latin1 character set):

root# mysqldump -u root -p --xml mylibrary > /tmp/mylibrary.xml
Enter password: ******

Exporting with mysql in Batch Mode

The universal tool mysql can be used to execute SQL commands in batch mode and store the results in a text file. In contrast to mysqldump, mysql is distinguished in that the resulting file is actually more or less readable—for human beings, that is. (However, there is no attempt to make the file suitable for a later reexportation.)

> mysql -u root --password=xxx --batch "--execute=SELECT * FROM TITLE;"
     databasename > output.txt

The SQL command is passed with the option --execute, where the entire option must be placed in quotation marks. In the resulting text file output.txt the columns are separated by tab characters. The first line contains the column headings.

Exchanging Rows and Columns

If a table has many columns, then mysql returns very long rows, which can be difficult to interpret. In this case, instead of --batch you can use the option --vertical. Then each record is divided over several lines, where in each row only one data item is given (this command must be typed on one line).

> mysql -u root --password=xxx --vertical
   "--execute=SELECT * FROM titles;" mylibrary > c:\tmp\test.txt

The resulting file looks like this:

*************************** 1. row ***************************
titleID: 1
  title: Linux, 5th ed.
subtitle: NULL
edition: NULL
  publID: 1
  catID: 1
  langID: NULL
    year: 2000
    isbn: NULL
comment: NULL
*************************** 2. row ***************************
titleID: 2
  title: Definitive Guide to Excel VBA
subtitle: NULL
edition: NULL
...

Generating HTML Tables

If instead of --batch the option --html is used, then mysql generates an HTML table with column headers (see Figure 10-1). However, the resulting file does not contain any HTML headers. Note as well that mysql does not deal with coding of characters outside of the 7-bit ASCII character set:

> mysql -u root --password=xxx --html
    "--execute=SELECT * FROM titles;" mylibrary > c:\\test.html

XML Exporting with mysql

Like mysqldump, mysql is also capable of creating XML files. The XML file contains no character set information, although one supposes that the latin1 character set is used (alas, not Unicode). The following command must be input in a single line:

> mysql -u root --password=saturn --xml
    "--execute=SELECT * FROM titles;" mylibrary > c:\test.xml

Astonishingly, the format of the resulting XML file is different from that of the file obtained by mysqldump; that is, it is apparently stored internally using a different conversion code.


International Customization, Character Sets

In the default setting of the binary distribution of MySQL (which is included in most Linux distributions), MySQL delivers error messages in English, uses the latin1 character set (ISO-8859-1), and sorts texts according to the Swedish rules. For applications in the English-speaking world this default setting is as it should be. (The peculiarities of the Swedish sorting rules have to do with characters outside of the 7-bit ASCII character set and do not affect the sorting of normal English text.)

Remark

The information in this section relates to MySQL 4.0. It is in particular with respect to character sets that MySQL 4.1 is to differ significantly: Instead of a character set for the entire server, each table, each column of each table, can be assigned its own character set. Furthermore, MySQL 4.1 offers (finally) support for the Unicode character set. Further information on the new character set functions of MySQL 4.1 can be found in Appendix B of this book.
Click To expand
Figure 10-1: A table generated with mysql

Error Messages in Other Languages

If you wish to have your MySQL server deliver its error messages in a language other than English, say German or French, then you have merely to set the option language in one of the configuration files. The language selected is also used for entries in the error logging file hostname.err.

MySQL currently supports about twenty languages. A glance at the directory mysql\share\ (Windows) or /usr/share/mysql (Linux) will tell you their names. If, for example, you find German error messages more to your liking than the English ones, then make the following change in the MySQL configuration file:

#in windows\my.ini or /etc/my.cnf
[mysqld]
language = german
Tip

Do not forget that you must restart the server for the changes to take effect. If MySQL does not find the file with the error messages, you can also give the entire path name with language, as, for example (under Windows), D:\Program Files\mysql\share\german

Selecting the Character Set and Sort Order

In MySQL the character set is selected with the option default-character-set, which is set in the MySQL configuration file. If this setting is missing, then the default is latin1 (ISO-8859-1).

Which character set is active and which character sets are otherwise available can be determined from the variables character_set and character_sets:

SHOW VARIABLES LIKE '%char%'
Variable_name    Value
-------------------------------------------------------------------------
character_set    latin1
character_sets   latin1 big5 czech euc_kr gb2312 gbk latin1_de sjis
                 tis620 ujis dec8 dos german1 hp8 koi8_ru latin2 swe7
                 usa7 cp1251 danish hebrew win1251 estonia hungarian
                 koi8_ukr win1251ukr greek win1250 croat cp1257 latin5
                 convert_character_set

Default Sort Order

Just to make matters confusing, with MySQL the choice of character set also influences the sort order. And since MySQL comes from Sweden, with latin1 the sort order is Swedish! Other SQL functions as well that are case-sensitive are influenced by the character set.

It is aggravating that the MySQL documentation offers no description of the available character sets. Their names are not always helpful, since they sometimes refer to the characters, sometimes to the sort order, and sometimes to both. (For example, german1 denotes the character set latin1 with German sort order, but latin1_de denotes the character set latin1, this time with the somewhat different German telephone directory order.)

Changing the Character Set and Sort Order

If youprefer a different sort order, you must change the option default-character-set in the MySQL configuration file (in the following example this is done for the German sort order). You will find a *.conf file for each supported character set in the directory mysql/share/charsets or /usr/share/mysql/charsets. If MySQL cannot find this directory, you must specify the location with the option character-sets-dir.

#in windows\my.ini or /etc/my.cnf
[mysqld]
character-sets-dir = Q:/mysql/share/charsets
default-character-set = german1

For the settings to become effective, the server must be restarted.

Remark

In MySQL 4.0, the setting of the character set holds for all tables and all clients. It is impossible to set the sort order for each table (or database). In particular, you as MySQL user have no say in the sort order: Only the MySQL administrator can change it. (In other words, if MySQL is running at the site of an ISP, you will have to make do with the sort order in force there.) The situation will become more flexible with MySQL 4.1; see Appendix B.

Resetting Indexes

After a change in the character set, you must reset all the table indexes. If you are using MyISAM tables, this is no problem. With myisamchk the indexes can be rather quickly regenerated. The required command looks like this:

root# myisamchk --recover --quick --set-character-set=german1 \
       mysqlpath/data/*/*.MYI

You must, of course, adapt the desired character set and the path to the database files to your own circumstances. On my test system the complete command looks like this:

root# myisamchk --recover --quick --set-character-set=german1 \
      /var/lib/mysql/*/*.MYI
Pointer

While myisamchk is being executed, the server must not make any changes in the database. (The easiest way to ensure this condition is to stop the server.) Further information on myisamchk is to be found later in this chapter.

It is somewhat more difficult to reset the indexes for InnoDB tables. Here you must delete the affected indexes with ALTER TABLE name DROP KEY and then replace them with ALTER TABLE name ADD KEY (column).

Client Configuration

The MySQL server informs its clients as to which character set it is using. However, the problem sometimes occurs that the client (e.g., mysql) needs the character set table but cannot find it. In most cases, the problem can be solved with the correct setting of character-sets-dir in the mysql group of the MySQL configuration file:

#in Windows\my.ini or /etc/my.cnf
[mysql]
character-sets-dir = Q:/mysql/share/charsets

If you are working under Windows with Connector/ODBC, you must copy the character set files into the directory C:\mysql\share\charsets.(You may need to create this directory.) Connector/ODBC is not capable of evaluating the MySQL configuration files, and therefore always looks in this predefined place for the character set files.)

Tests with Some Character Sets

This section shows the results of some tests with various character sets. If you are working under Windows, you must execute the SQL commands in a program that uses the Windows character set (such as the MySQL Control Center). The program mysql, on the other hand, will not do, since the command window uses a DOS character set for compatibility reasons.

Default Character Set latin1 with Swedish Sort Order

That Swedish sort order is used becomes apparent when you sort records alphabetically. The table test_order_by in the test database exceptions contains 255 records, where id contains the code, and a_char the symbol for the corresponding code. (The result of the following query is displayed, exceptionally, not by columns, but by rows, with entries separated by commas. Various special symbols and unrepresentable symbols were deleted from the result.)

Note that certain variants of the letter A are arrayed between A and B, while the letters Å, Ä, and Ö appear after Z (gotta love that Swedish sort order!):

USE exceptions
SELECT a_char FROM test_order_by
WHERE id>31
ORDER BY a_char
!, ", #, $, %, &, ', (, ), *, +, ,, -, ., /, 0, 1, 2, 3, 4, 5, 6, 7,
8, 9, :, ;, <, =, >, ?, @,
A, a, À, Á, Â, Ã, À, á, â, ã, B, b, C, c, Ç, ç, D, d, …, …, E, e, È,
É, Ê, Ë, è, é, ê, ë, F, f, G, g, H, h, I, i, Ì, Í, Î, Ï, ì, í, î, ï,
J, j, K, k, L, l, M, m, N, n, Ñ, ñ, O, o, Ò, Ó, Ô, Õ, ò, ó, ô, õ, P,
p, Q, q, R, r, S, s, T, t, U, u, Ù, Ú, Û, ù, ú, û, V, v, W, w, X, x,
Y, y, Ü, Ý, ü, ý, Z, z, [, Å, å, \, Ä, Æ, ä, æ, ], Ö, ö, …,
f,…, Š, …, Ž, …, š, …, ž, Ÿ, …, …, …, ÿ, …

The character set influences how MySQL deals with character strings. Compare the result of the following command with the result using the character set german1, shown a few lines later:

SELECT 'a'='A', 'a'='ä'
 'a'='A' 'a'='ä'
---------------------
    1       0

The consequence for sorting is that a and A are viewed as equivalent for sorting purposes, while a and ä are not. In fact, the Swedish town of Rämmen comes alphabetically after Rzeszów, in Poland.

Tests with default-character-set=german1

In what follows, the SQL test commands will be executed once again. We see that a and ä are viewed as equivalent for the purpose of sorting. Furthermore, the sort position of some letters has been changed:

!, ", #, $, %, &, ', (, ), *, +, ,, -, ., /, 0, 1, 2, 3, 4, 5, 6, 7, 8,
9, :, ;, <, =, >, ?, @,
A, a, À, Á, Â, Ã, Ä, Å, Æ, à, á, â, ã, ä, å, æ, B, b, C, c, Ç, ç, D, d, E, e,
È, É, Ê, Ë, è, é, ê, ë, F, f, G, g, H, h, I, i, Ì, Í, Î, Ï, ì, í, î, ï, J, j,
K, k, L, l, M, m, N, n, Ñ, ñ, O, o, Ò, Ó, Ô, Õ, Ö, Ø, ò, ó, ô, õ, ö, ø, P, p, Q, q,
R, r, S, s, …, T, t, U, u, Ù, Ú, Û, Ü, ù, ú, û, ü, V, v, W, w, X, x, Y, y, …, …, Z, z, Š, …, š, …, ž, Ÿ, ÿ
SELECT 'a'='A', 'a'='ä'
 'a'='A' 'a'='ä'
-------------------
    1       1


Logging

The term "logging" generally denotes the recording of each change in a database. (As you will see in the course of this section, there are other items that can be recorded as well.) Thus, if a MySQL client program executes the command

INSERT INTO mydatabase.mytable (col1, col2) VALUES (1, 'xy')

then MySQL changes the table mydatabase.mytable accordingly. If logging is activated, then the command itself is also saved in the logging file.

Pointer

In the MySQL documentation you will find information on the subject of logging:

http://www.mysql.com/doc/en/Log_Files.html

Why Logging?

There are various goals that can be achieved through logging:

  • Security: It is not possible to execute backups uninterruptedly. Thus, for example, if you make a backup every night, and then the next day your hard drive crashes, all that day's changes would be lost. If you have a logging file (naturally, on a different hard drive), then you can recreate the lost data.

    Logging files also help in determining who has used the database when, at what times an unusually large number of failed attempts to access MySQL were made, etc.

  • Monitoring: Normally, you know only the current state of the data. If your tables contain TIMESTAMP columns, then the time of the last change can easily be determined. But it is impossible to determine who made this change. If such information is important for your system, if it should be possible to find out who changed what data when, then you can activate an expanded logging protocol that will capture this additional information.

  • Optimization: Often, it is difficult to determine which queries are most burdensome to a database system. Therefore, MySQL offers the possibility to log all queries whose execution takes a particularly long time.

  • Replication: Replication denotes the process of synchronizing databases on several computers. For replication it is required that logging be carried out on the master computer. (Details on replication follow in the next section of this chapter.)

  • Transactions: Logging is also carried out with transaction-capable tables (such as InnoDB). In this case, logging files ensure that completed transactions can be executed even after a crash of the database. (Details on transaction logging for InnoDB tables appear later in this chapter.)

Based on the various purposes that it fulfills, MySQL supports various types of logging, which are presented in this section. As long as logging is activated, the logging files are stored in the same directory in which the directories of the various MySQL databases are located. (There is, however, the possibility of using various options to change the location. If you value maximum speed, the logging files should reside on a different hard drive from that on which the database files are located.)

Drawbacks

In the default setting almost all logging variants are deactivated. (The only exception is the logging of errors.) The main reason for this deactivation is speed: Logging slows down the operation of MySQL considerably (only, of course, if data are altered frequently).

The second drawback is the tendency of logging files rapidly to consume more and more space on your hard drive. In particular, with database systems in which data are frequently altered, logging files often require significantly more space than the actual database.

Logging Changes

The logging of changes to a database can be carried out in two ways in MySQL: in text form (log-update) or in binary format (log-bin). The binary format has a number of advantages:

The logging file contains more information, such as the time of each change and the client thread ID number. (If you wish to know the users behind the ID numbers, you must activate an additional logging file with the option log, in which each login to the MySQL server is recorded.)

Transactions are handled correctly. (The commands of a transaction are recorded only after a COMMIT.)

  • The logging files are significantly smaller.

  • Writing the logging files is more efficient.

  • The logging files can be used as the basis for replication.

The only drawback to binary logging is that the logging files cannot be viewed with an ordinary text editor; however, they can be viewed with the auxiliary program mysqlbinlog.

Since the MySQL documentation considers text logging as obsolete and no longer recommends its use, this section concentrates on binary logging.

Remark

In principle, it is possible to use log-bin and log-update simultaneously. Then logging will take place simultaneously in text and binary modes. Of course, this will slow things down considerably.

Activating Binary Update Logging

Binary logging is activated by the line log-bin in the configuration file. MySQL must then be restarted so that the change can become effective.

#in /etc/my.cnf or Windows\my.ini
[mysqld]
log-bin

MySQL now generates two new files, hostname-bin.001 and hostname-bin.index, in the directory in which the database directories are located. (Instead of hostname, the name of the computer is used.) The file hostname-bin.index contains merely a directory of all logging files. (At each server start a new logging file is begun.)

MySQL also logs in hostname-bin.n all changes to the contents and structure of all databases (thus, for example, the generation of a new table, adding a new column).

MySQL logs only actual changes. SELECT and UPDATE commands that leave the data unchanged will not be logged.

Caution

In the case of text importation of data with LOAD DATA only the command itself is logged, not the changes to the database. The logging file can therefore be used for a backup only if the underlying import file is still available.

The logging files can be viewed with mysqlbinlog; they look somewhat like the following example (where we have broken long lines and provided some comments):

> mysqlbinlog saturn-bin.002
#at4
#030318 17:29:14 server id 1 log_pos 4
# Start: binlog v 3, server v 4.0.12-nt-log created 030318 17:29:14
#at79
#030318 17:32:50 server id 1 log_pos 79 Intvar
SET INSERT_ID=61;
#at107
#030318 17:32:50 server id 1 log_pos 107 Query
# thread_id=1 exec_time=0 error_code=0
USE mylibrary;
SET TIMESTAMP=1048005170;
INSERT INTO authors (authName) VALUES ('Hesse Hermann');
#at202
#030318 17:47:00 server id 1 log_pos 202 Query
# thread_id=3 exec_time=0 error_code=0
USE myforum;
SET TIMESTAMP=1048006020;
UPDATE 'users'
SET 'email'='testaccount@dummy.com'
WHERE 'userID'=8;

Starting New Logging Files

When you stop and restart the MySQL server, the server automatically begins to fill the next logging file with data (hostname-bin.002, .003, etc.). A change in the logging file can, of course, be effected without a server restart, and in fact, with the SQL command FLUSH LOGS or with mysqladmin flush-logs. Furthermore, MySQL automatically begins a new logging file when the current file exceeds size max_binlog_size (1 gigabyte by default).

A suitable time to begin filling new logging files is after the execution of a complete backup. (The new logging file can help later to reconstruct all changes that occurred after the backup.)

Logging files that are no longer needed should be deleted with PURGE MASTER LOGS TO. This command is described later in this chapter when we discuss replication.

Restoring a Database on the Basis of a Logging File

If, perish the thought, misfortune should visit you, then the reconstruction of your database begins with the last full backup to restore the database. Then you employ all your logging files made since the last backup. The SQL commands contained therein are simply executed with mysql. (Be careful to follow the correct order. The oldest logging file must be executed first.)

root# mysqlbinlog hostname-bin.031 | mysql -u root -p
root# mysqlbinlog hostname-bin.032 | mysql -u root -p
root# mysqlbinlog hostname-bin.033 | mysql -u root -p

Determining the Location of the Logging File

You can determine the location of the logging file yourself. (For reasons of security and speed the logging files should be located on a different hard drive from that on which the database resides.) For this you specify the file name with the option log-bin. MySQL extends the file name with .nnn (that is, with a three-digit running integer).

Please make sure that the MySQL server, which under Unix/Linux is usually executed as user mysql, has writing privileges in the specified directory (here /var/log/mysql):

#in /etc/my.cnf or Windows\my.ini
[mysqld]
log-bin = /var/log/mysql/updates

Logging and Transactions

If you employ a table type that supports transactions, you must use binary logging. With traditional logging you will find BEGIN, COMMIT, and ROLLBACK simply ignored (that is, not logged). All other SQL commands will be logged at once. Therefore, even those SQL commands withdrawn by ROLLBACK will remain in the log.

On the other hand, with binary logging, changes will be logged only when the transaction has been completed with COMMIT.

Logging Files for Various Table Types

The logging procedure described in this section works for all table types (it is independent of type). However, some table types require their own logging files. Information on logging files for InnoDB appears later in this chapter.

Errors, Logins, and Slow Queries

Logging Errors

MySQL automatically logs each launch and shutdown of the MySQL server as well as all server error messages in the file hostname.err in the database directory. (Instead of hostname, the actual computer name is used.) Unfortunately, server start problems are not logged if they result from invalid configuration settings in my.ini or my.cnf.

There is no option whereby this error logging can be prevented. With the option log-error,however, you can set the name and location of the file:

#in /etc/my.cnf or Windows\my.ini
[mysqld]
log-error=/var/log/mysql/mysqlerrorlog

Logging Logins and Operations (General Query Log)

With the option log you can enable the logging of each connection to MySQL as well as every command. If you do not specify a file name, then MySQL creates the file hostname.log in the database directory, where instead of hostname, the actual computer name is used. The resulting file is not for the purpose of restoring data but for keeping track of which users are looking at and changing what data, etc.

#in /etc/my.cnf or Windows\my.ini
[mysqld]
log

The following lines show a small section of such a log:

MySql, Version: 4.0.12-nt-log, started with:
TCP Port: 3306, Named Pipe: MySQL

Time            Id Command  Argument
030318 16:15:33 1  Connect  root@127.0.0.1 on
                1  Init DB  mylibrary
030318 16:15:38 1  Query    select * from authors
030318 16:16:01 2  Connect  root@127.0.0.1 on
                2  Init DB  mylibraryodbc
                2  Query    SHOW TABLE STATUS FROM
                            'mylibraryodbc'
030318 16:16:03 2  Init DB  mylibraryinno
                2  Query    SHOW TABLE STATUS FROM
                            'mylibraryinno'
030318 16:16:06 2  Query    SHOW COLUMNS FROM
                            'mylibraryinno'.'authors'
                3  Connect  root@127.0.0.1 on
                3  Query    SET SQL_SELECT_LIMIT=DEFAULT
                3  Init DB  mylibraryinno
                3  Query    SHOW TABLES
                3  Query    SHOW FIELDS FROM 'authors'

In comparison to update logging, here all commands are logged immediately, as soon as they are passed to the server. This holds as well for transactions, indpendent of whether they are later completed with a COMMIT. In contrast to update logging, all commands are actually logged, even those that do not alter any data.

Logging Slow Queries

If MySQL is becoming painfully slow due to the burden of complex queries, it is often useful to undertake an analysis to find out which SELECT queries are actually causing the greatest delays. To do this, use the option log-slow-queries. Then MySQL will create a logging file with the name hostname-slow.log.

All queries whose execution takes longer than ten seconds will be logged. You can change this default time limit by setting the variable long_query_time. By setting the option log-long-format you can have all queries logged, even those that must be carried out without the use of an index (e.g., SELECT * ... WHERE txtcolumn LIKE '%abc%'):

#in /etc/my.cnf or Windows\my.ini
[mysqld]
log-slow-queries
long_query_time=5
log-long-format

Administration of Logging files

Logging files are like children: It is easier to create them than to take care of them, and furthermore, they grow on you. And just as children will take over your life if you aren't careful, your logging files will very quickly fill even the largest hard drive if you don't take prophylactic measures. So here is a bit of advice:

  • Do not log more than is absolutely necessary.

  • Use binary format if possible for update logs.

Update Logs

With mysqladmin flush-logs or the SQL command FLUSH LOGS, MySQL closes the currently active logging file and begins a new update logging file (with a new running number).

You have unrestricted access to all update logging files with the exception of the currently active file. You can thus move these files to another directory, compress them, and even delete them (for example, if they exceed a certain age or if they are older than the last complete backup). Under Unix/Linux you can develop a cron job to automate this task.

Logins and Slow Queries

With logging files for logins and operations (option log) and for slow queries (option log-slow-queries) there is a problem in that there is only one file, which gradually grows larger and larger. The MySQL documentation recommends that you simply rename these files while MySQL is running and then execute mysqladmin flush-logs or FLUSH LOGS:

root# cd logpath
root# mv hostname.log hostname.log.old
root# mv hostname-slow.log hostname-slow.old
root# mysqladmin flush-logs

The *.old files can now be compressed and eventually deleted.

Error Logging

Alas, neither mysqladmin flush-logs nor FLUSH LOGS has any effect on the logging file for error messages (hostname.err). Therefore, it is impossible to manipulate this file while MySQL is running. If this file becomes too large (which is seldom the case), you must stop MySQL, rename the file, and then restart MySQL.


Replication

Replication makes it possible to synchronize two or more MySQL servers running on different computers.

Different database systems employ differing methods of establishing replication. If you are familiar with replication from another database system, you should not expect MySQL to exhibit the exact same properties.

MySQL currently supports master/slave replication exclusively. There is one master system (read/write). This system is responsible for all changes to data. Additionally, there are one or more slave systems (read-only) on which, perhaps after abrief delay, exactly the same data are available as on the master system.

The exchange of data between the master and slaves is accomplished via binary logging files belonging to the master. The slaves remain in contact with the master and synchronize their databases by taking the requisite SQL commands from the logging files.

Replication functions even when the computers are running under different operating systems. (For example, the master can be running under Linux and a slave under Windows.)

Pointer

In addition to the information of this section, in Chapter 18 you will find a reference to all SQL commands for running replication (Chapter 18 provides an overview), and in Chapter 19 there is a reference to all mysqld options related to replication.

Further information can be found in the MySQL documentation, the section "Replication in MySQL":

http://www.mysql.com/doc/en/Replication.html

Why Replication?

There are two reasons that argue in favor of setting up a replication system: security and speed.

Security

Thanks to replication, your database is available on several computers. If a slave computer goes off line, the entire system can continue to run without interruption. (A new slave system can later synchronize itself.)

If the master computer goes off line, then the preservation of the data can be taken over by a slave computer. Alternatively, the entire system can be reconfigured so that a slave computer takes over the role of the master. In either case, though, the entire system is available as read-only.

If you are considering replication for reasons of security only, you should also consider a RAID system, whereby the contents of two (or more) hard drives are synchronized. A RAID system, however, protects only against a hard-drive crash, and not against an operating system crash, power outage, or the like.

Note that replication can also be used as a substitute for conventional backups. (Thanks to replication, the backup is always current. If you also require a conventional backup, you can execute it on the slave computer without involving the master computer.)

Speed

If the speed of a database system is limited primarily by many read-only queries (and not by a large number of alterations to the data), then a replication system can gain you great savings in time: The expensive queries are divided among several slave systems, while the master system is used exclusively or primarily for updates. (Of course, part of the theoretical increase in speed is lost due to the increased communication overhead.)

Please note that you can gain speed only if the programming of the client is compatible with your system. The client programs must divide your queries according to a load-balancing procedure (or simply at random) among all available slave systems. MySQL itself provides no mechanism for this purpose.

If your interest in replication is motivated by performance problems, you should consider alternative performance-enhancing measures, in particular, better hardware (in the following order: more RAM, faster hard drive, a RAID system, a multiprocessor system).

Limitations

  • MySQL currently supports replication only in the form of a master/slave system (one-way replication). All changes in data must be carried out on a single master system. The slave systems can be used only for database queries (read-only).

    It is not currently possible for a slave system to take over the role of the master automatically if it should go out of service (fail-safe replication). Thus replication can produce a system that is secure against breakdowns for database queries, but not for alterations to the data. Fail-safe replication is planned for MySQL 5.0.

    It is also impossible to synchronize changes to several systems (multiple-master replication). That would lead to problems with AUTO_INCREMENT values. It is therefore impossible, for example, to execute changes in a MySQL database on a notebook computer and later bring these into balance with the master system on another computer.

  • The replication system does not work properly with several SQL commands:

    • RAND: In MySQL 3.23, random numbers generated with RAND cannot be replicated. Every copy of the data contains a different value. To circumvent this problem, you may use RAND(n), where n is a pseudorandom parameter of the current timestamp value of your client program. Starting with MySQL 4.0, replication works with RAND( ) even without a parameter. ◦

    • User variables: In MySQL 4.0, SQL commands that contain their own variables (@varname) are not correctly replicated. This restriction will be removed with version 4.1. ◦

    • LOAD DATA INFILE: Data balancing works only if the imported file on the server is available during replication.

  • It is desirable that the same MySQL version be running on both master and slave computers. Replication generally functions correctly when the version number of the slave is higher than that of the client.

  • On master and slave systems the same character set should be used (setting default-character-set).

Setting Up the Replication Master System

This section describes the preparatory work for setting up a replication system on the master computer. Note, please, that this introduction does not show the only way to proceed. There are even several ways for transferring the start data to the slave. (One variant will be shown a bit later in this chapter.)

This section and the next assume that the database mysql, too, is to be replicated with its access privileges. This is usually a good idea, so that all users who are permitted to read data from the master system will be able to read data from the slaves with the same access information.

This method of proceeding is, however, burdened with some drawbacks. Everyone who is permitted to change the master is now permitted to do the same to the slaves. However, changes to data should fundamentally be made to the master, for otherwise, replication falls apart. If you thus wish to exclude data alteration on the slaves, you (as database administrator) must exclude the mysql database from replication (binlog-ignore-db=mysql), and instead, manage the mysql databases separately for the master and all of the slaves. However, that causes synchronization problems. For example, if a user obtains a new password on the master, the changed password must be entered on all the slaves.

Setting Up the Replication User

The first step consists in setting up a new user on the master system that is responsible for communication between master and client. The user name is irrelevant, and in this section we will use the name replicuser.

This user requires the Replication Slave privilege for access to the binary logging files. Instead of slavehostname, specify the complete computer name or IP number of the slave computer. For security reasons you should use as password a combination of characters that is in use neither in the operating system nor in the database:

GRANT REPLICATION SLAVE ON *.* TO replicuser@slavehostname IDENTIFIED BY 'xxx'

If youhave in mind to use the commands LOAD TABLE FROM MASTER and LOAD DATA FROM MASTER, then you must also grant replicuser the privileges Select, Reload, and Super. These commands are conceived primarily as aids for the MySQL developer and experts; they can help in setting up and managing a replication system.

If the replication system is to have several slaves, then you must execute GRANT for all the slave computers. Alternatively, you can permit access for all computers in the local system (e.g., replicuser@'%.netname'). This can simplify administration, though at the cost of introducing an unnecessary security risk.

On the slave system, test whether it is possible to establish a connection. (This has only the effect of ruling out possible errors that have nothing to do with replication.)

Shutdown

In setting up the slave (see the next section) you must specify the position in the logging files at which the slave is to begin reading. You can determine this position with the following command:

FLUSH TABLES WITH READ LOCK
SHOW MASTER STATUS
 File            Position Binlog_do_db  Binlog_ignore_db
-----------------------------------------------------------
 saturn-bin.005  79

You must note the information from the first two columns (file and position). If SHOW MASTER STATUS returns no result (empty set), then binary logging is not yet activated.

Next, the MySQL server must be shut down, for example, with mysqladmin shutdown or /etc/init.d/mysql stop), or under Windows, WinMySQLadmin.

Creating a Snapshot

Now you create a copy (called a snapshot) of all the databases. You will need this snapshot for the installation of the initial state of all the databases on the slave computers.

Under Windows use WinZip or another compression program; under Unix/Linux your best bet is to use tar. (If you have enough space on your hard drive, you may simply copy the database directory to another location.)

root# cd mysql-data-dir
root# tar czf snapshot.tgz database1/ database2/ database3/

With tar you cannot, unfortunately, use * to include all databases, because in the MySQL database directory there are usually many logging files that should not be installed on the slave system.

Server Configuration

In order for the MySQL server to be able to function as a replication master, you must associate a unique ID number with the server, using the option server-id. (Every computer in the replication system must be so designated.) Moreover, you must activate binary logging with the option log-bin if that has not yet been accomplished:

# master configuration
#in /etc/my.cnf or windows\my.ini
[mysqld]
log-bin
server-id=1

Then restart the server. (You can again use the server in normal fashion. All changes to the database will now be recorded in the binary logging file. As soon as a slave system goes on line, it will automatically synchronize its database based on the logging files.)

Tip

A brief description of additional master options can be found in the mysqld reference in Chapter 19. For example, you can restrict the replication to individual databases (option binlog-do-db) or exclude several databases from the replication (binlog-ignore-db).

Setting Up the Replication Slave System

Setting Up the Databases (Snapshot)

If the MySQL server is already using replication that is no longer to be used but is to be replaced by a new configuration, then execute RESET SLAVE. With this, the server forgets the old replication configuration.

Now stop the slave server. If there were already databases there, move their files into a backup directory (better safe than sorry).

Then copy the database files of the snapshot into the database directory with WinZip or tar xzf). Make sure that the files can be read and written by the MySQL server. Under Unix/Linux you do this by executing chown -R mysql.mysql (Red Hat) or chown -R mysql.daemon (SuSE).

Configuration File

With the slave system as well, the configuration file must be changed a bit. With server-id each slave system also obtains a unique identification number. With master-host, master-user, and master-password you specify how the slave system is related to the master:

# slave configuration
#in /etc/my.cnf or windows\my.ini
[mysqld]
server-id=2
default-character-set = <as for the master>
innodb_xxx = <as for the master>

Now the slave system can be brought on line. Start the server. If problems arise, look at the error log (file hostname.err).

Remark

Please be sure that the same character set is specified in the configuration files of all the slaves (option default-character-set)asfor the master.

If InnoDB files were in the snapshot, then the configuration file of the slave must have exactly the same InnoDB options as that of the master so that the InnoDB tablespace files can be correctly recognized when the server is started.

The logging of all changes (i.e., update logging) is not necessary for the slaves and should be deactivated for speed optimization.

A short description of additional slave options can be found in the mysqld option reference in Chapter 19.

Starting Replication

To start replication between slave and master, execute the following command. The slanted text should be replaced by the configuration data of the master:

CHANGE MASTER TO
    MASTER_HOST =     'master_hostname',
    MASTER_USER =     'replication user name',
    MASTER_PASSWORD = 'replication password',
    MASTER_LOG_FILE = 'log file name',
    MASTER_LOG_POS =  log_offset

If the master is running on the computer saturn.log and all other specifications are the same as those of the previous section, then the command would look like this:

CHANGE MASTER TO
    MASTER_HOST =     'saturn.sol',
    MASTER_USER =     'replicuser',
    MASTER_PASSWORD = 'xxx',
    MASTER_LOG_FILE = 'saturn-bin.005',
    MASTER_LOG_POS = 79

If binary logging was not instituted before replication was set up, then specify MASTER_LOG_FILE = '' and MASTER_LOG_POS = 4.

First Test

With mysql create a connection to the master system and add a new data record to the table of your choice. Then use mysql to establish a connection to a slave system and test whether the new record appears there. If that is the case, then rejoice, for your replication system is working already. (Of course, you could also generate and then delete new tables and even entire databases. The slave system understands and carries out these commands as well.)

Take a look, too, into the logging file hostname.err on the slave system. There you should see entries on the status of the replication, for example in the following form:

030319 14:11:57 Slave I/O thread: connected to master
   'replicuser@saturn.sol:3306', replication started in log
'saturn-bin.005' at position 79

As a further test you can shut down the slave system, make changes in the master system, and then start up the slave system again. The databases on the slave system should be automatically synchronized within a couple of seconds.

Setting Up a Replication System with LOAD DATA

Areplication system can be set up much more easily if one makes use of the command LOAD DATA FROM MASTER.A couple of conditions must be satisfied first:

  • On the master, MyISAM tables must be used exclusively. (In MySQL 4.0.12, LOAD DATA does not work with InnoDB tables. This should change in future versions, but there is no concrete information in the MySQL documentation as to when that might be.)

  • The mysql database should not be replicated. (LOAD DATA ignores the mysql database. Therefore, this database must already exist on the slave system.)

  • The MySQL configuration file of the master should contain log-bin and a unique server-id setting.

  • The MySQL configuration file of the slave should also contain log-bin and a unique server-id setting.

Master: If these conditions are satisfied, then setting up the replication system is child's play. You set up the replication user on the master, to which you grant the privileges Select, Reload, and Super:

GRANT SELECT, RELOAD, SUPER, REPLICATION SLAVE ON *.*
  TO replicuser@slavehostname IDENTIFIED BY 'xxx'

If the databases on the master are already filled in, then the variables net_read_timeout and net_write_timeout should have their values increased. (The default value is 20. The variables should be reset to the default after the replication system has been set up.)

SET GLOBAL net_read_timeout=600
SET GLOBAL net_write_timeout=600

Slave: On the slave, set the host name and user name and password for the replication process:

CHANGE MASTER TO
    MASTER_HOST =     'saturn.sol',
    MASTER_USER =     'replicuser',
    MASTER_PASSWORD = 'xxx'

On the slave as well, net_read_timeout and net_write_timeout should be increased:

SET net_read_timeout=600
SET net_write_timeout=600
MASTER_USER =

The following command transfers all the databases and tables from master to slave and starts the replication system:

LOAD DATA FROM MASTER

If an error occurs in executing this command, then life becomes complicated: You must stop the slave server, delete all (partially) transferred database directories, and then begin again. Possible sources of the problem are too-small values of net_read_timeout and net_write_timeout (from master and/or slave) as well as the presence of InnoDB tables. LOAD DATA FROM MASTER looks at the configuration settings replicate_ignore_xxx, with which individual tables or entire databases can be excluded from replication (see Chapter 19).

Replication Viewed from Inside

The master.info file

At the initial startup of replication, the slave system will have the file master.in added to its database directory. In this file MySQL keeps track of which binary logging file is currently being used, to what point this file has been evaluated, how the master can be contacted (host name, user name, password), etc. This file is absolutely necessary for the operation of replication. The MySQL documentation naturally emphatically recommends that you not mess around with this file:

saturn-bin.007
265
saturn.sol
replicuser
saturn
3306
60
Remark

The content of master.info can also be determinedwiththeSQLcommand SHOW SLAVE STATUS. Changes can be carried out with CHANGE MASTER TO. Both of these commands will be described in the SQL reference in Chapter 18.

relay Files (Slave)

Once replication is running on the slave computer, the files relay-log.info, hostname-relay-bin.index, and hostname-relay-bin.nnn appear in the data directory. These files are created by a separate IO thread (subprocess) on the slave server, using a copy of the binary logging files on the master. The sole task of the IO thread is to copy these data from master to slave. A second SQL thread then executes the SQL commands contained in the logging files.

Excluding Databases and Tables from Replication

If you do not want all of the databases or tables of the master replicated, there are two ways of excluding some of them: You can exclude databases from logging in the master configuration file (binlog-ignore-db), or you can exclude databases and tables from replication in the slave configuration file (replicate-ignore-table, replicate-wild-ignore-table, replication-ignore-db).

Excluding Replication Temporarily (Master)

If you wish to execute an SQL command on the master that is not to be replicated on the slave, then first execute the command SQL_LOG_BIN=0 and then the command SET SQL_LOG_BIN=1.(For this, the Super privilege is necessary.)

Ending the Master and Slave Servers

Master and slave servers run independently of each other, and they can be stopped and restarted independently and in either order without loss of data. If the slave is unable to make a connection to the master, or to reconnect, then it attempts a connection every sixty seconds. As soon as the connection is reestablished, all outstanding changes are read from the binary logging files and executed on the slave. This works even if the slave was down for a long time and has a great deal of catching up to do.

Several Slaves, Replication Chains

One may have an arbitrary number of slaves, all accessing the same master. From the master's point of view, nothing is different (except for the additional burden of accesses).

In addition, MySQL offers the possibility of creating replication chains of the form A ® B ® C . Here B would be a slave with respect to A, but a master with respect to C. This generally increases the overhead and is therefore not worthwhile. However, a possible scenario is a slow network connection between A and B (say, one server is in Europe and the other in the USA) and a fast connection between B and C. On computer B, the configuration log-slave-updates must be used.

Replication and Transactions

Transactions are not executed on the slave systems until they have been terminated on the master by COMMIT. On the other hand, if a transaction on the master is terminated by ROLLBACK, then the affeced SQL commands are neither logged in the logging files nor executed on the slave system.

It is even possible to use a transaction-capable table format (InnodB) on the master system, while using an ordinary MyISAM table on the slave. Since transaction management takes place entirely on the master, it is unnecessary to have slave support on the slave. However, it is necessary to have the slave properly set up before replication begins. (If the tables in the start state come from a file snapshot, then they are in the same format as those of the master and must be explicitly transformed on the slave to MyISAM tables. Note that changes in the format of a table are replicated from master to slaves.)

Remark

The seldom used commands LOAD TABLE/DATA FROM MASTER function only for MyISAM tables. It may be necessary for you to transform InnoDB tables on the master beforehand into this format.

Client Programming

If the goal of a replication system is to ensure against system failure or to increase speed by dividing up the queries among several systems, then changes are necessary to the client as well.

In establishing the connection, a distinction must be made as to whether data are going to queried only or whether changes are required as well. If INSERT, UPDATE, or DELETE commands are to be executed, then a connection to the master system must be made.

On the other hand, if data are to be queried only, then the connection should be made to the server that is currently the least burdened. Since the client program as a rule has no way of determining this, the Connect function should randomly select a computer from a predetermined list of computer names or IP numbers and attempt to make a connection. If this does not succeed (because this server is currently unreachable), then Connect should make an attempt to connect to another server.

Random Server Selection

The following example in the programming language PHP assumes that you wish to read (and not alter) data and to select the server randomly to improve efficiency. To do this you define in mysqlhosts[ ] an array of all the server names (or IP addresses) and use rand(min, max) to select an element of the array.

After the connection has been established, the single query mysql_list_dbs is executed (corresponding to the SQL command SHOW DATABASES):

<html><head><title>test</title></head><body>
<?php
    $mysqlhosts[0]="venus.sol"; // list of all servers of the
    $mysqlhosts[1]="mars.sol";// replication system
    $mysqlhost=$mysqlhosts[rand(0,1)]; // select server randomly
    $mysqluser="user"; // user name
    $mysqlpasswd="xxx";// password
    $connID=mysql_connect($mysqlhost, $mysqluser, $mysqlpasswd);
    $result=mysql_list_dbs();
    echo "<p>Databases at $mysqlhost<p>\n";
    while($row = mysql_fetch_row($result)) {
        echo "<br /><i>$row[0]</i>\n"; }
?>
</body></html>

Crashproof Server Selection

The starting point for our second example is similar. However, now our goal is to make the connection process immune to a connection failure. The following lines demonstrate a possible procedure whereby at most ten connection attempts are made.

$tries=0;
while($tries<10 && !$connID) {
    $mysqlhost=$mysqlhosts[rand(0,1)];
    $connID = @mysqld endedysqluser, $mysqlpasswd);
    $tries++;
}
if(!$connID) {
    echo "<p>Sorry, no database server found.\n";
    echo "</body></html>";
    exit();
}


Maintenance of MyISAM Tables

As we have already mentioned in Chapter 5, MyISAM tables are stored in the files dbname/tablename.MYD (data) and dbname/tablename.MYI (indexes). This allows the simple copying and moving of tables and databases. However, such operations are allowed only if the server is not using the databases (the best procedure is to stop the server for this purpose).

In rare cases, it may be necessary to work with the MyISAM table files directly, such as to restore indexes or repair damaged files. This section offers the tools that you will need.

myisamchk

The command myisamchk is, in a sense, a universal tool for the maintenance of MyISAM tables. With this command you can accomplish the following:

  • Check the integrity of MyISAM tables.

  • Repair damaged MyISAM table files (e.g., after a power outage).

  • Release unused storage space in MyISAM files.

  • Recreate the index to MyISAM tables (for example, after a change in the sort order of the MySQL server).

Instead of myisamchk,you can also use the following SQL commands:

  • ANALYZE TABLE provides information about internal index management.

  • CHECK TABLE tests the table file for errors in consistency.

  • OPTIMIZE TABLE optimizes the use of storage space in tables.

  • REPAIR TABLE attempts to repair defective tables.

These commands currently work only for MyISAM tables. However, it is possible that their effectiveness will someday extend to other types of tables. The advantage of the SQL commands over myisamchk is that you do not need to worry about the MySQL server and myisamchk interfering with each other. The disadvantage is that the MySQL server must be running (which may be problematic after a crash), that under some circumstances not all errors can be corrected, that there are fewer options for control of the process, and that the SQL commands are somewhat slower in their execution.

An extension to myisamchk is the command myisampack, with which MyISAM tables can be compressed. In this way, a great deal of space can be saved. However, only read access to such tables is then possible. At the end of this section we shall have more to say about myisampack.

Pointer

A reference to all options of myisamchk and myisampack can be found in Chapter 19. Further information on the use of myisamchk, particularly for the repair of defective table files, can be found in the MySQL documentation:

http://www.mysql.com/doc/en/Table_maintenance.html

Using myisamchk

The syntax of the myisamchk command is as follows:

myisamchk [options] tablename1 tablename2 ...

The table names are given as complete file names, either with or without the ending *.MYI (but, surprisingly, not with *.MYD). Depending on the options specified, however, both MyISAM files, that is, name.MYD (data) and name.MYI (indexes), are analyzed or changed.

To check on the integrity of all tables in the database mydatabase you should execute the following command. (You must, of course, replace /var/lib/mysql with your actual database directory.)

root# myisamchk /var/lib/mysql/mydatabase/*.MYI

You can use myisamchk independently of the MySQL server (the server may be running, but it does not have to be). If the server is running, then mysqladmin flush-tables or the SQL command FLUSH TABLES must be executed first.

Caution

If myisamchk actually changes MyISAM files and not just checks them, it must be ensured that the MySQL server does not change any data during this time.

Therefore, you must execute if necessary the SQL command LOCK TABLES with mysql, followed by myisamchk, and then, finally, UNLOCK TABLES. You must not leave mysql during this time, since otherwise, the LOCKswould end.

Speed Optimization, Memory Usage

In the case of large tables, the analysis, and even more the repair, of tables is a very costly operation. The speed of myisamchk depends greatly on the amount of available RAM.

The memory usage of myisamchk is set by four variables. In the default setting, myisamchk requires about 3 megabytes of RAM. If you have more memory to squander, then you should raise the values of the appropriate variables, since then myisamchk will execute much more quickly for large tables. The MySQL documentation recommends the following values:

root# myisamchk -O sort=16M -O key=16M -O read=1M -O write=1M
...

Here x should represent about one-fourth of available RAM (e.g., 64 MB on a 256-MB computer).

Furthermore, for repairing database files myisamchk requires a huge amount of space on the hard drive (among other reasons, because a copy of the database file is first made). A copy of the file is placed in the directory specified by the environment variable TMPDIR.You can also specify this directory via --tmpdir.

Shrinking and Optimizing MyISAM Tables

The MyISAM table driver attempts, normally, to keep table files as small as possible. However, if you delete a large number of records from your tables or if you often carry out changes to records with columns of variable size (VARCHAR, xxxTEXT, xxxBLOB), then the optimization algorithm runs up against its limit. In the worst case, the database files are significantly larger than necessary. Moreover, the data are scattered throughout the file, which slows down access to the database.

The following command provides some assistance. It regenerates the database file and optimizes the index file in view of providing the speediest access to the database. With the option --set-character-set the character set is specified for the sort order. (You must be dealing with the same character set with which the MySQL server is running.) The effect of --check-only-changed is that only those tables are processed that were changed since the last processing by myisamchk. (For space reasons, the command has been split using a backslash over two lines.)

root# myisamchk --recover --check-only-changed --sort-index \
      --analyze --set-character-set=xxx databasepath/*.MYI
Tip

If myisamchk is unable to find the character set files, you must specify it explicitly with --character-set-dir.UnderWindows, the files can be found under a standard installation in C:\mysql\share\charsets.

Tip

If you would like to check the performance of myisamchk, create a test database and then delete at random about half of the data records: DELETE FROM testtable WHERE RAND()>0.5

Repairing MyISAM Tables

For me, this section is largely of a theoretical nature, because fortunately, I have thus far had no problems with corrupt MyISAM tables. Corrupted files can arise when the database is stopped by a power failure, when MySQL or the operating system crashes, or if MySQL or the MyISAM table driver contains errors (which is rather unlikely, however).

Damaged MyISAM files make themselves known in MySQL service by error messages like Index-file/Record-file/Table is crashed or No more room in index/record file. In such a case, myisamchk will not, of course, be able to work a miracle. Data that for some reason are no longer available or have been overwritten cannot be restored. However, myisamchk can repair the database to the extent that at least all other records can again be read:

root# myisamchk --recover --set-character-set=xxx databasepath/*.MYI

If you suspect that only the index file has been affected, then execute myisamchk with the additional option --quick (which is considerably faster). In this case myisamchk regenerates the index file.

In particularly difficult cases, that is, when myisamchk --recover fails, you can attempt recovery with --safe-recover. However, that will take much longer than --recover.

If MySQL is running with a character set other than the standard latin1, you must specify the character set for sorting with the option --character-sets-dir.

Restoring or Creating a New MyISAM Index

If you change the character set of the MySQL server (for which a restart is necessary), then you must generate new indexes for all your tables. The command for doing so looks like this:

root# myisamchk --recover --quick --set-character-set=xxx databasepath/*.MYI

Compressing MyISAM Tables (myisampack)

If you exclusively read (but do not change) large tables, then it is a good idea to compress your files. Not only does this save space, but in general, it speeds up access (since larger portions of the table can reside in the file buffer of the operating system):

root# myisampack databasepath/*.MYI

Although with myisampack the identifier *.MYI is specified for the index file, the command changes only the data file *.MYD.To uncompress compressed table files, you should execute myisamchk with the option --unpack.


Administration of InnoDB Tables

As we have described extensively in Chapter 8, InnoDB tables offer, in comparison to MyISAM tables, a number of additional functions (in particular, transactions and integrity rules). However, the use of InnoDB tables makes the administration of MySQL a bit more complex. This section provides some tips for managing InnoDB tables and for the optimal configuration of the MySQL server.

Pointer

A quick overview of the most important InnoDB parameters can be obtained with the command SHOW VARIABLES LIKE 'innodb%'.A summary of the InnoDB configuration parameters for the MySQL server can be found in Chapter 19.

tablespace Administration

While MyISAM files and indexes are stored in their own files within a directory with the name of the database (e.g., data/dbname/tablename.myd), all InnoDB tables and indexes are stored in a virtual file system, which in the InnoDB documentation is called the tablespace. The tablespace itself can be composed of a number of files.

You can use InnoDB tables without any particular configuration settings, in which case at the first start of the MySQL server the file ibdata1, of size 10 megabytes, is created for the tablespace and can be enlarged repeatedly by 8 megabytes as required.

Determining the Space Requirements of InnoDB Tables

The tablespace is more or less a black box that you cannot see into. For example, there is no command to return any sort of directory of the tablespace. The command SHOW TABLE STATUS does give information about how much space the individual InnoDB tables and their indexes require within the tablespace and how much space is available before the tablespace will have to be enlarged.

In what follows, only the relevant columns of the SHOW command are displayed. The storage of tables and indexes is managed in 16-kilobyte blocks, which is why SHOW always returns integer multiples of 16,384:

SHOW TABLE STATUS FROM mylibraryinno
 Name        Type    Data_Length  Index_Length  Comment
---------------------------------------------------------------------------
 authors     InnoDB  16384        16384         InnoDB free: 3072 kB . . .
 categories  InnoDB  16384        49152         InnoDB free: 3072 kB . . .
  . . .

Configuration of the tablespace Files

Where and in what size tablespace files are created is determined by the configuration parameters innodb_data_home and innodb_data_file_path. The former specifies the directory in which all InnoDb files are stored (by default the MySQL data directory), and the latter contains the names and sizes of the tablespace files. A possible setting in the MySQL configuration file is the following:

#in /etc/my.cnf bzw. Windows\my.ini
[myslqd]
innodb_data_home = D:/data
innodb_data_file_path = ibdata1:1G;ibdata2:1G:autoextend:max:2G

This means that the tablespace consists of the files D:\data\ibdata1 and D:\data\ibdata2.If these files do not exist, they will be created, of size 1 gigabyte each. If the InnoDB tables require more space while the server is running, then ibdata2 will be enlarged automatically in 8-megabyte increments (to a maximum of 2 gigabytes).

Remark

The autoextend attribute has been available since MySQL 4.0.2 and 3.23.50. In MySQL 3.23.n, the configuration of innodb_data_home and innodb_data_file_path is absolutely required, while since 4.0.0 it has been optional.

In the management of the tablespace files you should observe the following:

  • The MySQL server requires write privileges for the innodb_data_home directory so that it can create and alter the tablespace files.

  • As administrator you must enlarge the tablespace in good time (that is, before reaching the limits of a tablespace file). If the InnoDB driver determines in a transaction that the tablespace is full and cannot be enlarged any further, the transaction will be aborted with ROLLBACK.

Using Hard Drive Partitions Directly

InnoDB also offers the option of using an entire hard drive partition directly (that is, without a file system managed by the operating system). For this, instead of specifying the file name, you give the device name of the partition and append the exact size specification (newraw). The partition size must be an integer multiple of 1 megabyte. (The following example uses the Linux device notation.)

innodb_data_home_dir=
innodb_data_file_path=/dev/hdb1:61440Mnewraw

After the partition has been initialized by MySQL, you must stop the MySQL server and replace newraw by raw.(The former is necessary only if you are adding a new partition.)

innodb_data_file_path=/dev/hdb1:61440Mraw

The InnoDB documentation unfortunately contains no information about whether better performance can be obtained by the direct use of hard drive partitions (one suspects that it can) and if so, how much. I have been able to obtain little concrete information from the MySQL mailing list.

Enlarging the tablespace

In principle, one cannot enlarge individual tablespace files. (The one exception is the autoextend attribute, which, however, can be specified only for the last tablespace file.) To enlarge the tablespace, you must therefore add an additional file to innodb_data_file_path. The process looks in detail like this:

  • Stop the MySQL server.

  • If the size of the last tablespace file was variable due to autoextend, you must determine its actual size in megabytes (the number of bytes according to DIR or ls divided by 1,048,576). The resulting size must be specified in the innodb_data_file_path setting.

    If innodb_data_file_path does not yet exist in the configuration file, then previously, ibdata was used as the default file, and you must determine and specify its size.

  • Add one or more new files to innodb_data_file_path.

    All files must be located in the one directory (or in directories relative to it). If you wish to divide the tablespace files among several partitions, hard drives, etc., then you must specify an empty character string for innodb_data_home, and in innodb_data_file_path you must use absolute file names.

    Note that the order of the files specified up to now in innodb_data_file_path cannot be changed (and that of course, none of the previous files may be missing).

  • Restart the MySQL server. If the server does not detect an erroneous configuration, it will generate the new tablespace files. This process will also log any errors in the file hostname.err.

Let us suppose that the previous setting looks like the following and that ibdata2 has current size of 1904 megabytes:

innodb_data_file_path = ibdata1:1G;ibdata2:1G:autoextend

Younow want to increase the size of the tablespace to 4 gigabytes. The new setting must look like the following:

innodb_data_file_path = ibdata1:1G;ibdata2:1904M;ibdata3:1100MB:autoextend

Shrinking the Size of the tablespace

It is, unfortunately, impossible to make the tablespace smaller. If you delete large InnoDB tables or change them into another table format, the space within the tablespace is freed up, but the tablespace files cannot be made smaller. The only way to shrink them is by the following process:

  • Make a backup of all InnoDB tables with mysqldump.

  • Delete all InnoDB tables (DROP TABLE ... ).

  • Stop the MySQL server (mysqladmin shutdown).

  • Delete the current tablespace files (ibdata ... ). If you have enough space, it is, of course, more secure first to move the files into another directory.

  • Change innodb_data_file_path in the MySQL configuration file.

  • Restart the MySQL server. New tablespace files corresponding to the innodb_data_file_path setting will be created.

  • Recreate all your InnoDB tables from your backup files.

    Tip

    Unfortunately, there is no way of obtaining a list of all InnoDB tables. If you are managing a large number of databases, you must search through the backup for each database for InnoDB tables. It is simpler to make a backup of all databases. The re-creation of the databases is also easier. But this way of proceeding makes sense only if the MyISAM tables constitute a small portion of the data or when you are going to make a complete backup anyhow.

    An alternative is to transform the InnoDB tables temporarily into MyISAM tables. This is easy to do, but involves a great deal of internal processing, and therefore, it is particularly slow for large tables. Furthermore, problems can arise if the tables use InnoDB-specific properties (such as foreign key constraints).

    Some tips for maximizing efficiency in large block operations such as inserting tables will be given later in this chapter.

Copying, Deleting, and Moving InnoDB Tables

From your experience with MyISAM tables you are accustomed to the fact that (after a server shutdown) you can simply copy or move all files dbname.tablename. The MySQL server recognizes at restart which tables are to be found where. This is very practical for backups or making a copy of a table or database.

If you use InnoDB tables, none of this is possible. If you wish to copy a table, you must either create a new table and copy the data with INSERT ... SELECT, or you must make a backup of the table (mysqldump) and then create the table under another name.

Furthermore, caution is necessary with the *.frm files. These files give the structure of a table and are located in the relevant database directory (even with InnoDB tables!). The *.frm files and associated tables stored in tablespace must always be synchronized. You must not simply delete *.frm files. If you wish to delete a table, execute DROP TABLE; then the *.frm file will be deleted.

Making a Backup

There are several ways of making a backup of InnoDB tables:

  • The most elegant way is to use InnoDB Hot Backup. This auxiliary program makes the backup while the server is running, without blocking tables. However, InnoDB Hot Backup is a commercial program and is not to be had for free; see http://www.innodb.com/hotbackup.html.

  • Of course, you can always use mysqldump. However, the results are consistent only if the tables are not changed during the execution of the command.

  • If youare prepared to stop the MySQL server, you can simply copy the tablespace files. Note, though, that you must use the exact same innodb_data_file_path setting as well as all *.frm files. On the other hand, you do not require the transaction logging files. If the server was properly shut down, these files no longer contain any relevant data.

Moving the tablespace

According to the MySQL documentation, the tablespace files are independent of the operating system. Only the CPU's floating-point representation must be correct. If those conditions are satisfied, then the tablespace files can be moved without problem between, say, Windows and Linux. Of course, you must take care here, too, that the innodb_data_file_path setting is correct and that all *.frm files are moved as well. (In practice, it will usually be the case that both MyISAM and InnoDB tables are to be copied. For this, all database directories and all tablespace files are simply copied.

The InnoDB documentation contains no information as to whether the tablespace file format is dependent in any way on the version. Heikki Tuuri, the developer of the InnoDB table driver, has promised forward compatibility for all present and future versions. (It has always been a tradition with MySQL that database files be able to be moved without difficulty to a new version.)

Within MySQL 4.0.n I experience no problems with changing among numerous versions; however, I have not checked backward compatibility to earlier versions. In general, an occasional backup with mysqldump is a good idea. The backup file then exists in text format, which is immune against possible compatibility problems.

Logging Files

Transaction Logging

InnoDB logs all changes in logging files with the names ib_logfile0, ib_logfile1, etc. The purpose of these logging files is to make large transactions possible as well as to restore InnoDB data after a crash.

If MySQL is properly configured and there is sufficient memory, then most of the currently needed data should reside in RAM. To improve speed, changes to data are first made only in RAM, and not in the actual data files (that is, in the case of InnoDB, in the tablespace).

Only when a transaction is completed with COMMIT are the changes in data actually stored on the hard drive, and then first in the InnoDB logging files ib_logfile0, ib_logfile1, etc. The changed parts of the tablespace are only gradually transferred to the hard drive, all this for reasons of efficiency. If a crash occurs during these proceedings, then the tablespace can be restored with the help of the logging files.

The logging files ib_logfile0, etc., have two purposes. On the one hand, they satisfy the ACID condition of durability, so that transactions that have been carried out are not endangered even if there is a crash immediately after the transaction has been completed. On the other hand, the logging files enable transactions of almost unlimited size, even those for which it is not possible to hold all pending (but not yet confirmed) changes in RAM.

The InnoDB logging files are filled in order. When the last file is full, the InnoDB table driver begins writing data to the first logging file. Therefore, the entire size of all logging files limits the quantity of tablespace changes that can be temporarily stored before a COMMIT. The maximum size of all logging files is currently (MySQL 4.0.9) limited to 4 gigabytes.

The transaction logging files are necessary only while the MySQL server is running. As soon as the MySQL server has been properly shut down, these files are no longer needed. For example, if you make a backup with ibdata files, you do not need to copy the logging files.

Size and Location of the Logging Files

The proper dimensioning of the logging files has a great influence on the speed of MySQL/InnoDB. The location, size, and number of logging files are determined by the configuration parameters innodb_log_group_home, innodb_log_files_in_group, and innodb_log_file_size. By default, two logging files of size 5 megabytes each are created in MySQL's data directory.

The InnoDB documentation recommends that the total size of the logging files be about the size of the buffer (parameter innodb_buffer_pool_size, which is 8 megabytes by default). If the buffer is larger than the logging files, then it can happen that the InnoDB table driver will have to make a so-called checkpoint, involving temporary storage of uncommitted changes.

If you wish to change the location, size, or number of logging files, you must stop the MySQL server. Now delete the existing logging files ib_logfile n (only when you are sure that shutdown took place without error) and change /etc/my.cnf or Windows\my.ini. New logging files will be created at the subsequent restart of the MySQL server.

Tip

If you value maximal speed, then you should have the logging files stored on a different hard drive from that on which the tablespace files reside.

Remark

If the MySQL server finds logging files when it is started up that do not correspond to the innodb_log_ parameters, then the startup process is broken off. You will find error messages in the error logging file hostname.err. The creation of new logging files (even if there is no error) is also logged in hostname.err.

Logging Synchronization

The configuration parameters innodb_flush_log_at_trx_commit and innodb_flush_method tell when (how often) and how logging files are synchronized. For innodb_log_at_trx_commit there are three settings, which allow one to make tradeoffs between speed and security:

  • The default setting is 0. The data are written about once per second into the current logging file, and then the file is synchronized. (Writing means here that the data are passed to an I/O function of the operating system. Synchronization means that changes are actually physically written to the hard drive.)

    If there is a crash during the time between the COMMIT and the synchronization of the logging file, the transaction is lost and cannot be reconstructed at a later restart. Therefore, innodb_flush_log_at_trx_commit=0 is a strike against the ACID durability condition.

  • More secure is the setting 1. Now writing and synchronization take place with each COMMIT. The drawback is that if you make mostly small transactions, then the hard drive limits the number of possible transactions per second. (For a hard drive with 7200 revolutions per minute, that is, 120 per second, at most 120 transactions can be executed per second, a theoretical limit that is never reached in practice.)

  • The setting 2 is a good compromise. Here the writing takes place at each COMMIT, but the synchronization only about once per second. If the MySQL server crashes, then immediately terminated transactions are not lost (since the synchronization can take place after a crash). However, if the operating system crashes (power outage, for example), then transactions are lost as with setting 0.

The parameter innodb_flush_method determines whether the operating system function fsync (the default) or O_SYNC (setting O_DSYNC) is used for synchronizing the logging files. With many Unix versions, O_SYNC is faster.

Archive Logging

The transaction logging files are conceived only for internal data management, not for backups. If you require a record of all changes to data since a particular time (since the last complete backup), then you must use MySQL binary logging, which functions entirely independently of the InnoDB table driver. (See the discussion earlier in this chapter.)

InnoDB can, in principle, also carry out such logging. This type of logging is called archive logging in the InnoDB documentation. However, archive logging makes sense only if InnoDB is used independently of MySQL. The files ib_arch_log_n that appear in the data directory are a result of such archive logging. In them are logged, at the start of the MySQL server, the creation of new tablespace or transaction logging files. Then, however, archive logging is automatically shut off.

Should you wish for some reason to use archive logging, it can be activated with innodb_log_archive.

Tips for Speed Optimization

This section offers some tips for speed optimization. The information here is relevant only if you are using primarily InnoDB tables and are working with large data sets.

Pointer

Look at the section "Performance Tuning Tips" in the InnoDB documentation. In optimizing for speed there is also useful information in the InnoDB status information, which can be retrieved with SHOW INNODB STATUS.

Buffer Settings

Perhaps the most important parameter for influencing the speed of the InnoDB table driver is innodb_buffer_pool_size. This parameter specifies how much RAM should be used for temporary storage of InnoDB tables and indexes. The more such data is available in RAM, the less often access must be made to the hard drive in SELECT commands. By default, the InnoDB table driver reserves only 8 megabytes as a buffer. Larger values (the InnoDB documentation recommends up to 80 percent of main memory for a dedicated database server) can dramatically increase the speed of SELECT queries. (The total size of the transaction logging files should be as large as the buffer storage.)

Depending on the application, two additional parameters influence what data are stored temporarily in RAM: innodb_log_buffer_size sets the size of the buffer for transaction logging, while innodb_additonal_mem_pool_size determines how much space in RAM is reserved for various other information such as metadata on open tables. This buffer (by default 1 megabyte) should be enlarged if you are dealing with a large number of InnoDB tables.

Block Operations

If you are carrying out extensive block operations (such as importing a table with a million data records or changing from MyISAM to InnoDB format), you can speed up the process with a few tricks:

  • Use the setting SET unique_checks=0. Then no check is made whether the data of a UNIQUE column or the primary index column are actually unique. Of course, you should use this setting only if you are absolutely sure that there are, in fact, no duplicates!

  • Use the setting SET foreign_key_checks=0. With this setting you achieve that the integrity conditions are not checked. Of course, here, too, this setting should be used only if you are convinced of the integrity of your data (e.g., in restoring backup data).

  • Execute all the INSERT commands for a table as a single transaction. Usually, importation consists of countless INSERT commands, which by default (auto commit)are all carried out in separate transactions. With SET AUTOCOMMIT=0, all INSERT commands are collected into a single transaction. Keep in mind that this transaction must be confirmed with a COMMIT.

    This technique works only if the transaction logging files are large enough. Note that with large transactions, a ROLLBACK can be unusually long, even taking hours. Note as well that CREATE TABLE has the effect of COMMIT. Therefore, it is impossible to read in several tables in a single transaction.

Logging Settings

In the previous section we made reference to the possible settings of the logging parameters. Here is a brief summary:

  • innodb_flush_log_at_trx_commit=2 is useful is you wish to execute as many (small) transactions per second as possible and are prepared to lose the last few seconds of data in case of a crash.

  • Ideally, the logging files should be located on a different hard disk from those of the tablespace files.

  • Depending on the operating system, innodb_flush_method=O_DSYNC may speed up logging.


Server Tuning

Server tuning refers to the optimal configuration of the MySQL server so that it uses hardware as efficiently as possible and executes SQL commands with maximum efficiency.

Server tuning is worthwhile, as a rule, only if very large databases are involved (in the gigabyte range), many queries per second are to be processed, and the computer is serving primarily as a database server.

This section provides merely a first introduction to this topic and is restricted primarily to the correct configuration of the buffer storage and the use of the query cache. Note, however, that server tuning is only a component of the larger topic that perhaps might be called optimization of database applications. On this theme one could easily write an entire book, which would, among other things, answer the following questions:

  • What possibilities are there to optimize the database design in such a way that the most frequently used commands are executed with maximum efficiency? Were the optimal indexes set up? The correct database design is assuredly the most important and undervalued component of speed optimization. Fine tuning the server for a poorly designed database is like hitching a racehorse to a carriage with square wheels. (See also Chapter 5.)

  • What is the best hardware for the task (within a given price range)?

  • What is the best operating system (if there is a choice)?

  • What is the optimal table format (see also Chapter 8 on InnodB).

  • Can the burden of many SELECT queries be divided among a number of computers?

    Pointer

    Many questions will go here unanswered, not least because I myself have too little experience with very large database applications. Further information can be found in the MySQL documentation:

    http://www.mysql.com/doc/en/MySQL_Optimisation.html

    Awealthoftuning information is contained in the web sites, news, and mailing list contributions of the MySQL guru Jeremy Zawodny. Do a Google search or have a look at the following site:

    http://jeremy.zawodny.com/mysql/

Optimal Memory Management

MySQL reserves at startup a portion of main memory for certain tasks, such as a cache for data records and a location for sorting data. The size of this buffer is controlled by options in the configuration file and generally cannot be altered while the server is in operation. It can happen that MySQL leaves a great deal of RAM unused, even though there was sufficient memory available and MySQL could make use of it.

The setting of the parameters takes place in the mysqld section of the MySQL configuration file. Memory sizes can be abbreviated by K (kilobytes), M (megabytes), and G (gigabytes). The following lines clarify the syntax. (In MySQL 3.23 the syntax is set-variable = varname=n. This syntax is still allowed, but it is no longer necessary.)

#in /etc/my.cnf bzw. Windows\my.ini
[myslqd]
key_buffer_size = 32M

In the following, various important configuration parameters will be introduced (though not all of them by a long shot). Unfortunately, one cannot say which parameter settings should be changed and to what, it all depends heavily on the specific application. However, first attempts should include key_buffer_size and table_cache:

  • The parameter key_buffer_size (default 8M) tells how much storage is to be reserved for index blocks. The higher the value, the more rapid is table access to columns for which there is an index. On dedicated database servers it can make sense to increase key_buffer_size up to one-fourth of the available RAM.

  • The parameter table_cache (default 64) specifies how many tables can be open at one time. The opening and closing of table files costs time, and so a larger value of the parameter can increase parallel access to many tables. On the other hand, open tables cost RAM, and the number is also limited by the operating system. The number of tables open in MySQL can be determined with SHOW STATUS (variable open_tables).

  • The parameter sort_buffer (default 2M) specifies the size of the sorting buffer. This buffer is used in SELECT commands with ORDER BY or GROUP BY if there is no index available. If the buffer is too small, then a temporary file must be used, which is, of course, slow. The default value of 2 megabytes should suffice for many purposes.

  • The parameter read_buffer_size (formerly record_buffer, default 128K) specifies how much memory each thread reserves for reading sequential data from tables. The parameter should not be unnecessarily large, since this memory is required for each new MySQL connection (thus for each MySQL thread, not only once for the entire server). It is best to increase the parameter only when it is needed for a particular session with SET SESSION read_buffer_size=n.

  • The parameter read_rnd_buffer_size (default 256K) has an effect similar to that of read_buffer_size, except that it is valid for the case in which the records are to be read out in a particular order (as with ORDER BY ). A larger value can avoid search operations on the hard disk, which can slow things down considerably with large tables. As with read_buffer_size, read_rnd_buffer_size should be increased only as needed with SET SESSION.

  • The parameter bulk_insert_buffer_size (default 8M) specifies how much memory is reserved for the execution of INSERT commands in which many records are to be inserted simultaneously (such as INSERT ... SELECT ... ). This parameter can also be changed for individual connections with SET SESSION.

  • The parameter join_buffer_size (default 128K) specifies how much memory is to be used for JOIN operations when there is no index for the JOIN columns. (For tables that are frequently linked there should definitely by an index for the linking field. This will contribute more to speed efficiency than increasing this parameter.)

  • The parameter tmp_table_size (default 32M) specifies how large temporary HEAP tables can get. If this size is exceeded, then the tables are transformed into MyISAM tables and stored in a temporary file.

  • The parameter max_connections (default 100) gives the maximum number of database connections that can be open at one time. The value should not be unnecessarily high, since each connection requires memory and a file descriptor. On the other hand, persistent connections profit from a larger number of allowed connections, since then it is less frequent that a connection is closed and a new one has to be opened. (With SHOW STATUS you can determine max_used_connections. This is the maximum number of connections that were open simultaneously up to a particular time.)

    Tip

    You can get a look at the most important current settings with the command SHOW VARIABLES LIKE '%size%.
    Pointer

    If you use InnoDB tables, you should definitely look also at the InnoDB options; see the previous section of this chapter.

    A brief description of all available parameters of the MySQL server can be obtained by launching mysqld with the option --help. A morecomplete description is given in the MySQL documentation:

    http://www.mysql.com/doc/en/SHOW_VARIABLES.html

Query Cache

The query cache is a new function in MySQL 4.0. The basic idea is to store the results of SQL queries. If later this exact same query is to be executed, then the stored result can be used instead of having to search through all the affected tables.

The query cache is no panacea, though, for speed optimization. In particular, queries must be deleted from the query cache as soon as the underlying tables are altered:

  • The query cache is therefore useful only if the data change relatively seldom (thus many SELECT commands in relation to the number of UPDATE, INSERT, and DELETE commands), and it is expected that particular queries will be freqently repeated (which is frequently the case with web applications).

  • The SELECT commands must be exactly the same (including spaces and case), so that the query cache knows that they are, in fact, the same.

  • The SELECT commands cannot contain user-defined variables and cannot use certain functions, the most significant of which are RAND, NOW, CURTIME, CURDATE, LAST_INSERT_ID, HOST.

If these conditions are not satisfied, then the query cache SELECT queries will, in the worst case, slow the system down somewhat (due to the management overhead).

Activating the Query Cache

By default, the query cache is deactivated (due to the default setting query_cache_size=0). To activate the query cache, execute the following changes to the MySQL configuration file:

#in /etc/my.cnf or Windows\my.ini
[myslqd]
query_cache_size = 32M
query_cache_type = 1
query_cache_limit = 50K
# 0=Off, 1=On, 2=Demand

Now 32 megabytes of RAM is reserved for the query cache. In the cache are stored only SELECT results that require less than 50 kilobytes. (This avoids the situation in which a few large query results force all other results out of the cache.)

After a server restart, the cache is automatically active. For MySQL applications nothing changes (except that the reaction time to repeated queries is less).

Demand Mode

The query cache can also be run in demand mode. In this case, only those SELECT queries are considered that are executed with the option SQL_CACHE, as in SELECT SQL_CACHE * FROM authors. This mode is useful if you wish to control which commands use the cache.

No Temporary Storage of SQL Query Results

If you wish to prevent a SELECT command from using the active query cache (query_cache_type=1), then simply add the option SQL_NO_CACHE. This makes sense with commands about which one is certain that they will not soon be repeated and would therefore take up space unnecessarily in the query cache.

Turning the Query Cache On and Off for a Connection

You can change the mode of the query cache for a particular connection. Just execute SET query_cache_type = 0/1/2/OFF/ON/DEMAND.

Determining the Status of the Query Cache

If you wish to know how well the query cache is functioning, whether its size is well chosen, etc., then execute the command SHOW STATUS LIKE 'qcache%'. As result youreceive a list of status variables, whose meaning is briefly described in Table 10-1.

With FLUSH QUERY CACHE you can defragment the cache (which, the MySQL documentation states, makes possible improved memory usage, but it does not empty the cache). RESET QUERY CACHE deletes all entries from the cache.


ISP Database Administration

Up to now we have assumed that you have installed MySQL on your own computer and have unrestricted access to the server. And indeed, this is the usual starting point for every database administrator.

However, with MySQL the situation can be a bit different. Often, your database is located on the computer of an Internet service provider. There you have almost no administrative privileges. That is, the ISP administers MySQL on its own. (If you have a responsible ISP, it will automatically carry out backups of your databases, but don't count on it.) Nonetheless, you will have to carry out certain administrative tasks:

  • Create new databases (provided that the ISP permits this).

  • Execute backups. (Even if your ISP does this regularly, it is good to be able to be responsible for your own data.)

  • Execute a database upload. (For example, you have developed a database application on your own computer and now wish to transfer the completed database to the ISP computer.)

Working via telnet/ssh

Administration is at its simplest when your ISP provides you with telnet access (or ssh access, which would be more secure). Then you can log into the ISP's computer and use all of the commands introduced in this chapter. Needless to say, with commands such as mysqldump you can access only your own databases. But that should suffice. Formoving files between your local computer and that of the ISP, you can use ftp.

Unfortunately, not all ISPs offer their clients access via telnet or ssh (or only for a large additional fee), since this means extra work for the ISP and greater security risks.

Working via phpMyAdmin

Probably the most popular solution to this administrative problem is offered by phpMyAdmin, a collection of PHP scripts installed in a www directory on the ISP's computer. In principle, phpMyAdmin is suited for all the administrative tasks mentioned above. In practice, however, there is usually a problem.

For the execution of PHP scripts—and thus for phpMyAdmin as well—there is a time limit. If the execution of a script exceeds the allotted time, the script is automatically terminated. In most cases this limit represents a sensible protective measure against programmer error (such as infinite loops), but unfortunately, it makes the backup of a large database impossible.

Table 10-1: Query cache status variables
Variable name | Description
Qcache_queries_in_cache | Tells how many queries have results in the cache.
Qcache_inserts | Tells how many queries have been cached up to now. This value is generally greater than Qcache_queries_in_cache, since results are removed from the cache due to lack of space or changes to tables.
Qcache_hits | Tells how often a query was able to be answered directly from the cache.
Qcache_lowmem_prunes | Tells how many results were deleted from the cache because it was full. (If this value is high, that is an indication that the cache size is too small.)
Qcache_not_cached | Tells how many queries were not accepted into the cache (e.g., because a function like RAND or NOW or the option SQL_NO_CACHE was used).
Qcache_free_memory | Tells how much memory is currently free in the cache.

Pointer

The installation of phpMyAdmin on an ISP computer is discussed in Chapter 2, while the details of using it appear in Chapter 4.

Implementing Custom PHP Scripts for Administration

Instead of working with phpMyAdmin, you can, of course, program your own PHP scripts for administration and store them in a directory on your web site. For example, you can execute a backup of a database with a PHP file of the following design:

<?php system("/usr/bin/mysqldump --host=hostname --user=username ".
"--password=xxx dbname > backup.sql"); ?>

Instead of /usr/bin you may have to specify a different path (for example, /usr/local/bin). In some circumstances things will work without a path name being specified. As hostname you can use localhost if the web server and MySQL are running on the same computer.

After you have loaded the page via your web browser (and thereby executed the script), you can transfer the file backup.sql, which is located in the same directory as the script, to your computer, again via the web browser or FTP.

Remark

With an ISP the web server (and thus the PHP script as well) is usually executed under the account nobody or apache. This user usually has writing privileges in your directories. mysqldump works only if you have granted extensive write privileges for the directory in which the PHP script is located. For this the command chmod a+w directory would be necessary, but you might not be able to execute it without telnet/ssh.Instead, use your FTP client to set the directory's access privileges.

Conversely, for an upload you first transfer via FTP the file upload.sql generated on your computer into the directory in which your PHP administration scripts are located. Then you execute a script via your web browser according to the following pattern:

<?php system("/usr/bin/mysql --host=hostname --user=username ".
"--password=xxx dbname < upload.sql"); ?>

For the upload to succeed, the specified database must already exist. If that is not the case, you can first create the database with mysqladmin (which you also execute in the PHP script via system).

If you have previously created upload.sql with mysqldump, then the database at the ISP may not contain the tables defined in upload.sql.If necessary, you must insert some DROP TABLE IF EXISTS commands in upload.sql.

In comparison to phpMyAdmin, the advantage of this way of proceeding is that no time is lost in the transfer of the database files over the Internet. Instead, the upload file is read from the local computer, or the download file is written there. However, this does not change anything in regard to the time limit for PHP scripts. Of course, you can transport somewhat larger databases with the method presented here than with phpMyAdmin, but sooner or later you will find yourself nose to nose with the PHP time limit.

Pointer

Do not forget to secure access to the directory with your administrative scripts with .htaccess.Theuseof .htaccess is described in Chapter 2.

Custom Perl Scripts for Administration

What works with PHP works also, of course, with Perl, and often better. The advantage of Perl is that in the execution of CGI scripts there is often no time limit. (But this depends on the provider.)

In Perl, too, there is a system function for calling external commands. In the lines below there is also code added that evaluates and displays the return value of this function. The mysqldump and mysql commands look exactly the same in the PHP scripts presented above. Even the preparatory work is identical:

#!/usr/bin/perl -w
use CGI qw(:standard);
use CGI::Carp qw(fatalsToBrowser);
print header(), start_html("Backup");
if(system("/usr/bin/mysqldump --host=hostname --user=username " .
          "--password=xxx dbname > backup.sql")) {
  print p(), "failed", end_html(); }
else {
  print p(), "done", end_html(); }


Part III: Programming

Chapter List

Chapter 11: PHP — Fundamentals and Programming Techniques
Chapter 12: PHP — Library Management
Chapter 13: PHP — Discussion Forum
Chapter 14: Perl
Chapter 15: Java, JSP, JDBC
Chapter 16: C and C++
Chapter 17: Visual Basic, C#, ODBC


Chapter 11: PHP - Fundamentals and Programming Techniques

Team LiB

Chapter 11: PHP — Fundamentals and Programming Techniques

Overview

The letters PHP stand for PHP Hypertext Preprocessor (a recursive abbreviation typical of Unix-world shenanigans). PHP is a script programming language for HTML pages. The code embedded in an HTML file is executed by the server. (The Active Server Pages of Microsoft also follow this plan.)

When a programming language is brought into play with MySQL, that language is almost always PHP. The reason for this is that MySQL offers almost ideal conditions for building dynamic web sites: simple deployment, high speed, unbeatable price (free).

This chapter provides general programming techniques. The following two chapters demonstrate two rather long application examples. (The examples can be tested on line at www.kofler.cc.)


Introduction

This chapter assumes that MySQL and PHP are correctly installed on your test system and that everything functions as it should. (Chapter 2 discusses installation.)

Pointer

All the example programs for this chapter are available for download at www.apress.com in the downloads section.

Acompact reference to all MySQL functions in PHP can be found in Chapter 20.

PHP Compatibility Problems

The examples in this book were tested with PHP versions 4.2 and 4.3. Most of the examples should work with PHP 4.1 as well.

However, if you use PHP 4.0 or an earlier version, you will have difficulties with all the examples that use the superglobal fields $GLOBALS, $_GET, $_PUT, $_COOKIE, $_SESSION, etc. (The term superglobal means that the variables are available in called functions and do not have to be passed as parameters.) The content of these fields is explained in Table 11-1.

Table 11-1: Superglobal fields (available since PHP 4.1)
Variable | Content
$GLOBALS | Contains the names of all variables that are currently globally valid.
$_SERVER | Contains variables of the HTTP server (generally from Apache).
$_GET | Contains variables that were passed with HTTP-GET. (This includes variables that are declared in the web address, e.g., http://my.firm/script.php?variable=123.)
$_POST | Contains variables arising from a form that were passed with HTTP-POST.
$_COOKIE | Contains cookie variables.
$_FILES | Contains variables that were passed with HTTP-POST that have information about files.
$_ENV | Contains environmental variables of the operating system.
$_REQUEST | Contains all variables from $_GET, $_POST, $_COOKIE as well as variables that were passed via URL character strings (that is, all variables that can be manipulated by the user and are considered insecure).
$_SESSION | Contains session variables ($_SESSION is available only after a call from start_session or session_register).

Access to the data of superglobal fields takes place in the example programs of this book in a code block of the following pattern. The function array_item that we have defined serves to initialize variables without an error message even if no data were passed. (A PHP page for the evaluation of a form can be called by another page, so that the form data are then lacking.)

function  array_item($ar, $key) {
  if(array_key_exists($key, $ar)) return($ar[$key]);}
$sqlType            = array_item($_REQUEST, 'sqlType');
$authID             = array_item($_REQUEST, 'authID');
$formSearch         = array_item($_POST, 'formSearch');
$formSubmitTitle    = array_item($_POST, 'formSubmitTitle');

For these examples to work in PHP 3.n or 4.0, you must comment out these instructions. Variables like sqlType and authID are then automatically initialized if corresponding data are passed.

This type of data transfer often represents a security risk, for which reason since PHP 4.1 it has been available only in a particular configuration (variable register_globals in php.ini). Since PHP 4.2 this direct passing of global variables is deactivated by default (register_globals=Off ).

Pointer

Further information on security in PHP applications and the use of superglobal fields can be found at the following addresses:

http://www.zend.com/zend/art/art-oertli.php
http://www.php.net/manual/en/reserved.variables.php
http://www.php.net/manual/en/language.variables.external.php
http://www.php.net/manual/en/security.registerglobals.php
http://www.php.net/manual/en/security.variables.php

PEAR

Direct access to MySQL by PHP can take place in two different ways:

  • by functions directly integrated into PHP, all of which begin with mysql_

  • by functions from the PHPExtension and Add-on Repository (PEAR for short)

This book considers only the mysql_ functions. PEAR offers better portability if the database system is later changed, but is not yet very widespread. (The PEAR database functions are comparable to DBI for Perl.)

Pointer

Further information on the use of the PEAR database functions can be found at the following addresses:


Establishing a Connection to the Database

The Connection

To create a connection, one usually uses the PHP function mysql_connect, to which three parameters are passed: the computer name (host name) of the MySQL server, the MySQL user name, and the password. If MySQL is running on the same computer as the PHP script (that is, on localhost), then the computer name does not have to be given:

$connId = mysql_connect("localhost", "username", "xxx");

This function returns an identification number for the connection. This number will be needed in the future only if you have more than one connection to MySQL open. (As long as there is only one connection to MySQL, this is the default connection. The ID number thus does not have to be given in calling various mysql_xxx functions.)

Optional Parameters

Beginning with PHP 4.1, you can specify with a fourth parameter whether in multiple execution of mysql_connect with the same connection data, simply a link to the existing connection should be passed (false, default behavior) or a new connection should be established (true). The form mysql_connect($host, $name, $pw, true) is required if you maintain several separate connections.

Beginning with PHP 4.2, you can pass client flags in a fifth parameter. A possible constant is MYSQL_CLIENT_COMPRESS (if the data exchange is to take place in compressed form):

$connId = mysql_connect("localhost", "username", "xxx",
    new_link, client_flags);

Problems with Establishing the Connection

If problems occur in establishing the connection, the variable connId will contain the value FALSE. Moreover, mysql_connect sends an error message to the web server, so that in the resulting HTML document a rather unattractive error message appears for you to behold. To avoid this error message it is necessary to place the @ character before mysql_connect. (In general, this character prevents error messages from being displayed in calls to PHP functions.)

If you would like to supply the PHP code for the connection with a readable error message, you might write code something like the following:

$connId = @mysql_connect("localhost", "username", "xxx");
if ($connID == FALSE) {
  echo "<p><b>I regret to inform you that a connection to the database
    cannot be established at this time.
    Please try again later. Perhaps you will have better luck.</b></p> \n";
  echo "</body></html>\n"; // close HTML Document!
  exit(); // end PHP Script
}

Selecting the Default Database

As soon as a connection is established, you can use various mysql_xxx functions to execute SQL commands and do a lot of other neat stuff as well. To avoid having to specify the desired database over and over, you can use mysql_select_db to select the default database (mysql_select_db corresponds to the SQL command USE databasename):

mysql_select_db("mylibrary");

Specifying the MySQL user Name and Password in an Include File

You should always avoid storing a user name and password in plain text in a file that is accessible over the web. Of course, visitors to your site should in principle never obtain a glimpse of the source text of PHP files, since the PHP code is evaluated by the web server and is no longer visible in the resulting HTML document. But configuration errors have been known to occur by which the PHP file in raw form is revealed to a web surfer.

An additional security risk is that the file might be read not via HTTP but via anonymous FTP. (That would also be the result of a configuration error. The directory for anonymous FTP should be completely separate from that with HTML files. Yet such configuration errors occur rather frequently.)

Thus to avoid allowing strangers to tumble onto your MySQL password too easily (but also so as not to have to write this information in every PHP script, which would entail a great deal of work if the MySQL password were to change), the MySQL login information is usually stored in its own file. For the examples of this section the password file has the name mylibraryconnect.inc.php, and it looks like this:

<?php
  // file general/mylibraryconnect.inc.php
   $mysqluser="user";        // user name for MySQL access
   $mysqlpasswd="xxx";       // password
   $mysqlhost="localhost";   // name of the computer on which MySQL is running
?>

The best place to store this file depends on the configuration of the web server. On my web site I have created a directory htdocs/_private/. The directory _private is secured by .htaccess. Therefore, loading the file directly with http://www.kofler.cc/_private/mylibraryconnect.inc.php is possible only if the HTTP user name given in .htaccess and the associated encrypted password stored in the authentication file are given.

In order to discover my MySQL password, the PHP configuration of the ISP of my web site would have to be defective. Furthermore, you would have to guess the name of the directory _private and break the password protection via .htaccess. It is rather unlikely that all of these conditions could be satisfied.

Caution

Be absolutely certain that the include file cannot be accessed by anonymous FTP.

Be sure as well that all include files end with *.php. Thus do not use, say, name.inc, but name.inc.php. This ensures that the file will be executed by the PHP interpreter during an HTTP access in every case (for example, if an attacker guesses the .htaccess user name and associated password).

Be sure that it is impossible via anonymous FTP to display a PHP file on your web server in such a way that this file can then be read over the web server. If there is an FTP directory called incoming, then this directory must also not be accessible over the web server (for example, as http://www.mysite.com/ftp/incoming). If that were the case, then an attacker could write a simple PHP script that reads your include file and reads out the contents of the variables. (To do this, the attacker would have to know the name of the include file and names of the variables, but they just might be guessed. Often, examples are taken directly from various books.)

We return now to the PHP file in which the connection to MySQL is to be established. There an include must be used to load the file with the password information. If you assume, for example, that the files intro.php and mylibrary-connect.inc.php are located in the directories

/www/user1234/htdocs/php-examples/general/intro.php
/www/user1234/htdocs/_private/mylibraryconnect.inc.php

then the include instruction in intro.php must look as follows:

// file intro.php
include("../../_private/mylibraryconnect.inc.php");
$connID = @mysql_connect($mysqlhost, $mysqluser, $mysqlpasswd);

Depending on how the directories containing the PHP scripts and the include file are located relative to each other, you will have to change the path information in the include command. The two periods (..) indicate that a subdirectory of the current directory is to be used.

Remark

In the example files to this book, the include file is located in the same directory as the script files. The include instruction therefore contains only the file name (that is, include("mylibraryconnect.inc.php")). If you wish to secure the example files, you must move the file mylibraryconnect.inc.php into a secure directory and then alter the include instruction.

Building the Connection in the Include File

If an include file is used, then one should make full use of its advantages. If you wish to create a connection to MySQL from several different PHP files, it makes sense to store all of the code for creating the connection together with error checking in the include file.

The following model assumes that the function mylibrary_connect is executed before any HTML headers or the like are created in the PHP script file. Of course, other ways of proceeding are possible, but take care that the resulting HTML document is complete (that is, inclusive of HTML header and with closure of all open HTML tags):

<?php
// file general/mylibrary-connect.inc.php
function  connect_to_mylibrary() {
  $mysqluser="useruser"; // user name
  $mysqlpasswd="xxx"; // password
  $mysqlhost="localhost"; // name of the computer of which MySQL is running
  $connID = @mysql_connect($mysqlhost, $mysqluser, $mysqlpasswd);
  if ($connID) {
    mysql_select_db("mylibrary"); // set default database
    return $connID;
  }
  else {
    echo "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.0//EN\">
       <html><head>
       <title>Sorry, no connection . . . </title>
       <body><p>Sorry, no connection to database  . . .    </p>
      </head></html>\n";
    exit(); // terminate PHP interpreter
    }
  }
?>

Example

The following miniscript (see also Figure 11-1) makes use of the function called connect_to_mylibrary introduced above to establish a connection to MySQL, and it then determines the number of data records stored in the table titles. (The two functions mysql_query and mysql_result are introduced in the following sections.)

Click To expand
Figure 11-1: A simple PHP example
<!-- php   /file general/intro.php
<?php
 include("mylibraryconnect.inc.php");
 $connID = connect_to_mylibrary();
?>
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN">
<html><head><title>PHP Programming, intro sample</title>
</head><body>
<?php
  $result = mysql_query("SELECT COUNT(*) FROM titles");
  echo "<p>Table <i>titles</i> contains ",
   mysql_result($result, 0, 0),
   "  records.</p>\n";
?>
</body></html>

Persistent Database Connections

PHP provides two functions for creating a connection to MySQL. One is the function mysql_connect, which we have already discussed, and the other is the function mysql_pconnect. The advantage of mysql_pconnect is that an attempt is made to reactivate an earlier connection with the same host name, user name, and password. This proceeds more quickly than establishing a completely new connection. In such a case, the connection is called persistent. (Establishing a connection to MySQL usually is rapid in any case, and thus the time savings resulting from a persistent connection are not enormous. On the other hand, with other database systems a persistent connection is absolutely necessary.)

Persistent connections are possible only if PHP is used as a module of the web server (and not in the CGI versions).

Persistent connections are managed by a thread. They are reused when the combination of host name, user name, and password coincides with an existing (but not currently in use) connection.

Regardless of how your web server is configured, you can always use mysql_pconnect. If recycling a connection is impossible, then a new connection is established. (Note that the ability to use persistent connections can be stopped in php.ini with the variable mysql.allow_persistent.)

Caution

The term "persistent" might lead one to think that the MySQL connection remains established when a PHP script is changed. That is not the case. In fact, it is quite the opposite.

As a rule, after a change from one PHP page to another, a different MySQL connection is used (which has opened another script).

Therefore, you may not depend on any connection-specific information (such as the current database) surviving a page change. In particular, you cannot execute SQL transactions over several pages.

Disadvantages of Persistent Connections

The small speed advantage achieved by persistent connections is alas paid for with some drawbacks:

  • On a heavily visited web server, mysql_pconnect leads to very many connections being opened to MySQL and then not being closed (so that they can be used again). This state of affairs consumes considerable memory; the number of connections is limited by max_connections (default setting is 100).

    Before you change max_connections in the MySQL configuration file, you should read about tuning. For example, with max_connections the number of file descriptors that mysqld needs increases. (The MySQL variable max_used_connections informs you about the maximum number of simultaneous connections that have occurred thus far. The content of the variable can be retrieved with SHOW STATUS.)

  • All settings that were carried out in a persistent connection hold for the next page that this connection uses. This can cause unwanted side effects.

  • If the PHP code of a page is aborted on account of an error, then temporary tables remain; already started transactions remain uncompleted (until they are either confirmed or rolled back by additional SQL commands, which might belong to an entirely different context); tables locked by LOCK remain locked.

  • Do not rely on temporary tables being automatically deleted at the end of a script. Temporary tables are automatically deleted at the end of a connection to MySQL, but the life of a persistent connection is of uncertain length, and as a rule, is much longer than the time during which a script is executed. Delete your temporary tables explicitly at the end of a script with DROP TABLE.

    Pointer

    An extensive discussion of the advantages and drawbacks of persistent connections can be found at the following address:

    http://www.mysql.com/newsletter/2002-11/


Executing SQL Commands

To execute an SQL command you pass it as a character string to the function mysql_query. If the command is not to be applied to the current database, then you can use mysql_db_query to provide the name of the intended database. With each of these functions the ID number of the connection (that is, the return value of mysql_connect) can be passed as the optional last parameter if there is more than one connection to MySQL:

$result = mysql_query("SELECT COUNT(*) FROM titles");
$result = mysql_db_query("mylibrary", "SELECT COUNT(*) FROM titles");

With mysql_query, any type of SQL command can be executed: queries with SELECT; changes to data with INSERT, UPDATE, and DELETE; changes to the database structure with CREATE TABLE; and so on.

Remark

An SQL command may not be terminated with a semicolon. It is not possible to give several commands at once. If you wish to execute several commands, then you must execute mysql_query once for each command.

Return Value ($result)

If an SQL command can be correctly executed, then mysql_query returns a nonzero value. If a query was involved, then, the return value of mysql_query is a reference to a PHP resource (for example, a character string of the form "Resource id #2". This return value can then be inserted into various other functions (for example, mysql_fetch_row) to evaluate the individual fields of a table. In the examples of this chapter the return value is usually stored in a variable with the name result. (Working with SELECT results will be described in detail in the next section.)

On the other hand, if an SQL command cannot be executed, then mysql_query returns the result FALSE (i.e., 0). Moreover, an error message is displayed, which you may suppress by executing mysql_query with the @ prefixed. (The cause of the error can be determined by evaluating mysql_errno and mysql_error. Further information on error evaluation can be found further below.)

Metainformation on Query Results

After INSERT, UPDATE, and DELETE commands (and all other commands that change data records), you can determine with mysql_affected_rows how many records were changed. This function is also helpful after CREATE TABLE ... SELECT ... for determining how many records were inserted into the new table.

Moreover, you can determine with mysql_insert_id after an INSERT command which AUTO_INCREMENT value was used for inserting the last new record:

$n = mysql_affected_rows(); // number of changed records
$new_id = mysql_insert_id(); // ID number of the last  AUTO_INCREMENT  record

On the other hand, if you have executed a SELECT query with mysql_query, you can use the functions mysql_num_rows and mysql_num_fields to determine the number of resulting data records and columns, respectively:

$rows = mysql_num_rows($result); // number of records
$cols = mysql_num_fields($result); // number of columns
Remark

Please note the various parameters that the functions we have been discussing take:

  • In the case of mysql_affected_rows and mysql_insert_id no parameter should be given. (Optionally, the ID number of the MySQL connection can be given, that is, $connID,ifyou follow the nomenclature of the examples of this chapter.)

  • On the other hand, mysql_num_rows and mysql_num_fields expect as parameter the ID number of the query (that is, $result).

Releasing Query Results

PHP stores query results until the end of the script. If you wish to release these results sooner (for example, because you wish to execute a large number of queries in a script and do not wish to use more memory than necessary), you can release the query result early with mysql_free_result. This is particularly to be recommended when there is a loop in your script that executes SQL queries:

mysql_free_result($result);


SELECT Queries

Evaluation of SELECT Queries

If you execute a SELECT query with mysql_query, then as result you obtain a reference to a table with rows rows and cols columns:

$result = mysql_query("SELECT * FROM titles");
$rows = mysql_num_rows($result);
$cols = mysql_num_fields($result);

This holds as well for two special cases:

  • If the query returns only a single value (for example, SELECT COUNT(*) FROM table), then the table has only one row and one column.

  • If the query does not return any result at all (for example, if there are no records that correspond to a WHERE condition), then the table has zero rows. This case can be identified only by evaluating mysql_num_rows.

Only if the SQL query was syntactically incorrect or there was a problem with communication with MySQL will you receive no result at all (that is, the return value of mysql_query is FALSE).

Access to Individual Table Fields

The simplest, but also the slowest, access to individual fields of a table is offered by mysql_result. You simply provide the desired row and column numbers in two parameters. (Numbering begins with 0, as with all PHP MySQL functions.) Instead of the column number, you can give the column name (or the alias name if in the SQL query you have worked with AS alias):

$item = mysql_result($result, $row, $col);

It is considerably more efficient to evaluate the result by rows. There are four functions for this:

$row = mysql_fetch_row($result);
$row = mysql_fetch_array($result);
$row = mysql_fetch_assoc($result); // since PHP 4.0.3
$row = mysql_fetch_object($result);

  • mysql_fetch_row returns the record in the form of a simple array. Access to the columns is accomplished with $row[$n].

  • mysql_fetch_array returns the record in the form of an associative array. Access to the columns is accomplished with $row[$n] or $row[$colname] (thus, for example, $row[3] or $row["publName"]). The column name must be given in case-sensitive form.

  • mysql_fetch_arraymysql_fetch_assoc (available since PHP 4.0.3) also returns an associative field, which can be read out in the form $row[$colname].In contrast to mysql_fetch_array, here it is not allowed to pass the column number as parameter.

  • mysql_fetch_object returns the record as an object. Access to the columns is accomplished with $row->colname.

A feature that all of these functions have in common is that with each call, the next record is automatically returned (or FALSE if the end of the list of records has been reached). If this given order is to be altered, then the currently active data record can be changed with mysql_data_seek:

mysql_data_seek($result, $rownr);

Column Names and Other Metainformation

If you have processed valid queries and wish to display the results, what you need are not only the data themselves, but also metainformation about the nature of the data: the names of the columns, their data types, and so on.

The function mysql_field_name returns the name of the specified column. The function mysql_field_table tells in addition from what table the data come (this is important in queries that collect data from several tables). The function mysql_field_type gives the data type of the column in the form of a character string (for example, "BLOB")). The function mysql_field_len gives the maximum length of a column (especially of interest with the data types CHAR and VARCHAR):

$colname = mysql_field_name($result, $n);
$tblname = mysql_field_table($result, $n);
$typename = mysql_field_type($result, $n);
$collength = mysql_field_len($result, $n);

Yet more detailed information about a column is given by mysql_field_flags and mysql_fetch_field: The function mysql_field_flags returns a character string in which the most important attributes of the column are given (for example, "not_null primary_key"). The properties are separated by a space. Evaluation proceeds most simply with the function explode.

On the other hand, mysql_fetch_field returns an object that partially provides the same information as the character string of mysql_field_flags. Evaluation proceeds in the form colinfo->name, colinfo->blob, etc. (A complete table of all object properties can be found in the API reference in Chapter 20.)

$colflags = mysql_field_flags($result, $n);
$colinfo = mysql_fetch_field($result, $n);

Displaying SELECT Results as a Table

The following example (see also Figure 11-2) shows how the result of a simple query (SELECT * FROM titles) can be displayed in an HTML table. The code for table output resides in the function show_table, which is defined in the file mylibraryconnect.inc.php. Note in particular that all character strings are translated into correct HTML code by the PHP function htmlentities. This function ensures that even character strings with HTML special characters like < and > are correctly represented.

The resulting table has the absolute minimum of bells and whistles. To be sure, background color and the like could yield a more cogent presentation, but that is not the theme of this book. (Any book on HTML will inform you of the necessary HTML tags that you would have to add to the code.)

Click To expand
Figure 11-2: The result of a simple query displayed as a table
Tip

Independent of questions of layout, you should take one further detail into consideration: In the display of numbers, right justification often gives a clearer presentation than left justification. You can determine the data type of each column and set the justification accordingly.

// file general/mylibraryconnect.inc.php
// displays the result of a query as an HTML table
function show_table($result)
{
  if(!$result) {
   echo "<p>Error in SQL statement.\n"; return;
  }
  $rows = mysql_num_rows($result);
  $cols = mysql_num_fields($result);
  if($rows>0) {
   echo "<table border=1>";
   echo "<tr>";
   // column headings
   for($i=0; $i<$cols; $i++) {
  echo "<th>", htmlentities(mysql_field_name($result, $i)),
     "</th>";
   }
   echo "</tr>";
    // table content
   while($row = mysql_fetch_row($result)) {
     echo "<tr>";
     for($i=0; $i<$cols; $i++) {
       echo "<td>", htmlentities($row[$i]), "</td>";
     }
     echo "</tr>\n";
   }
   echo "</table>";
  }
}
?>

The function show_table is activated by the PHP script select.php. There a simple SQL query is executed and the result variable result passed to show_table:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN">
<?php   // file general/select.php
  include("mylibraryconnect.inc.php");
  $connID = connect_to_mylibrary();
?>
<html><head>
<meta http-equiv="Content-Type"
    content="text/html; charset=iso-8859-1" />
<title>PHP Programming, SELECT sample</title>
</head><body>
<?php
  // execute SQL query
  $result = mysql_query("SELECT * FROM titles");
  // display metainformation
  $rows = mysql_num_rows($result);
  $cols = mysql_num_fields($result);
  echo "<p>\$result = $result\n",
    "<br />mysql_num_rows(\$result) = $rows\n",
    "<br />mysql_num_cols(\$result) = $cols</p>\n";
  // display HTML table with result
  echo "<p>\n";
  show_table($result);
  echo "</p>\n";
?>
</body></html>
Pointer

When a query returns many results, it is not a good idea simply to display all the results. It is better to divide the results over several pages. Furthermore, the user should be given the option of jumping from one page to the next (and back again). An example of a pagewise representation of SELECT queries can be found in Chapter 12.

SELECT Queries with mysql_unbuffered_query

Since PHP 4.0.6 you can execute SELECT queries with mysql_unbuffered_query (instead of with mysql_buffered_query). This has the effect that the query result is not immediately moved into the PHP interpreter's memory, but only when needed (that is, when individual result records are to be output). This yields the following consequences:

  • The data remain on the server until they are fetched by the PHP script. This means that during this time, resources on the MySQL server are blocked. (In exchange, the PHP interpreter needs to reserve only a small amount of memory for intermediate storage.)

  • You must read in the found records one at a time (forward only) with mysql_fetch_row, mysql_fetch_object, mysql_fetch_assoc, mysql_fetch_array. The functions mysql_fetch_result and mysql_data_seek, on the other hand, are not available. Already read records cannot be read a second time, since only a single record is ever located in the PHP interpreter's memory.

  • You may not execute a new SQL command for the current MySQL connection until you have read all the records of the SELECT result. If you do not wish to do this, either you will need a second connection to the MySQL server or you will have to release the SELECT result explicitly with mysql_free_result($result).

  • You can determine the number of found records only by running through all the result records. You cannot use mysql_num_rows.

  • mysql_unbuffered_query is designed only for SELECT queries (and not for commands that do not return results, such as INSERT and DELETE).

mysql_unbuffered_query is especially suited for cases in which large SELECT results are to be processed record by record, without the reservation of much client memory for the purpose. Since it is in the nature of PHP pages generally to process small quantities of data, and since the execution time of PHP code is limited (setting max_execution_time in php.ini), there are only a few situations in which the function offers real advantages over mysql_query.


Changes to Data (INSERT, UPDATE, DELETE)

There are no MySQL functions included in PHP for changing data. All alterations must be made with mysql_query in the form of SQL commands (INSERT, UPDATE, DELETE, etc.). This means as well that all data (BLOBs included) must be transmitted as character strings.

Inserting Linked Data Records

If you insert records into tables with AUTO_INCREMENT, you can determine the last AUTO_INCREMENT value generated by MySQL with the function mysql_insert_id, which we have already mentioned.

The following lines show how a new book title, together with its two authors, can be stored in the mylibrary database. Please note that three ID values must be temporarily stored, which then must be inserted together in the rel_title_author table. It is assumed that the publisher of this book (here Addison-Wesley) is already stored in the database with publID=1.

mysql_query("INSERT INTO titles (title, publID, year)
    VALUES ('A Guide to the SQL Standard', 1, 1997)");
$titleID = mysql_insert_id();
mysql_query("INSERT INTO authors (authName)
    VALUES ('Date Chris')");
$author1ID = mysql_insert_id();
mysql_query("INSERT INTO authors (authName)
    VALUES ('Darween Hugh')");
$author2ID = mysql_insert_id();
mysql_query("INSERT INTO rel_title_author (titleID, authID, authNr)
    VALUES ($titleID, $author1ID, 1),
           ($titleID, $author2ID, 2)");
Pointer

In practice, that is, when data are input interactively, the storing of new input is somewhat more complicated. For example, it must be checked whether an author already exists. If so, the ID of that author must be stored in rel_title_author. If not, a new author must be inserted. (Or was there perhaps a typo? To minimize this possibility, similar names in sound or appearance are displayed and offered as alternatives.) An extensive example will be presented in the next chapter.

Determining the Number of Changed Records after INSERT, UPDATE, and DELETE

The function mysql_affected_rows that we have already mentioned returns the number of records that were changed by an INSERT, UPDATE,or DELETE command.

Remark

mysql_affected_rows returns 0 if all records of a table were deleted with a DELETE command without a WHERE clause. mysql_affected_rows returns with an UPDATE command only the number of records that were actually changed. Records that were considered, say, with a WHERE clause but whose data agreed with the existing data are not counted. mysql_affected_rows is not designed for SELECT commands and returns 0 in such a case.

Obtaining Status Information After Changes

Due to the limitations of mysql_affected_rows, since PHP 4.3 there has been the function mysql_info. This function returns, after the following commands, a character string with status information on the MySQL server:

ALTER TABLE

CREATE TABLE ... SELECT

INSERT INTO

LOAD DATA

UPDATE

The character string contains, for example, information about duplicates and warnings. This is the same string that is displayed in mysql after the execution of the above-mentioned commands. The form of the string depends on the particular command (and could change with new versions of the MySQL server).

If, for example, you execute the command CREATE TABLE backup SELECT * FROM table, then mysql_info returns a string of the form "Records: 27 Duplicates: 0 Warnings: 0".


Character Strings, BLOBs, DATEs, SETs, ENUMs, and NULL

This section describes how to deal with various data types that in practice present a number of difficulties. Coping with NULL will also be covered. This section deals separately with storing, altering, and output of data.

Altering Data

To alter data in a database the corresponding SQL commands must be passed as character strings to mysql_query. In constructing such character strings, the rules are the same as for MySQL (see Chapter 18). That is, dates and times must be passed in the syntax prescribed by MySQL.

The starting point for the following discussion of various data types is the variable data, which contains the data to be stored. The contents of this variable should be formed into an INSERT command, which is stored temporarily in the variable sql. In the simplest case it works like this:

$sql = "INSERT INTO tablename VALUES('$data1', '$data2',  . . .    );

Of course, you can put sql together piece by piece:

$sql = "INSERT INTO tablename VALUES(";
$sql .= "'$data1', ";
$sql .= "'$data2', ";
   . . .
$sql .= ")";

  • Dates and Times: To format a date or time in accordance with the MySQL rules, you can summon to your aid the PHP functions date and strftime. If data are the result of user input, you should carry out the usual validation tests. (MySQL carries out only very superficial validation and is perfectly happy to store data that cannot possibly refer to a real date or time.)

  • Timestamps: PHP and MySQL timestamps are really the same thing, but they are formatted differently. In the case of PHP timestamps we are dealing simply with a 32-bit integer that gives the number of seconds since 1/1/1970. MySQL, on the other hand, expects timestamps in the form yyyyddmmhhmmss or yyyy-dd-mm hh:mm:ss.

    As a rule, timestamps are used to indicate the time of the last change. In this case you simply pass NULL, and MySQL takes care of the correct storage automatically:

    $sql .= "NULL";

    On the other hand, if you wish to store a PHP timestamp as a MySQL timestamp, then you should rely on MySQL FROM_UNIXTIME in the INSERT or UPDATE command:

    $data = time(); // data contains the current time as a Unix timestamp
$sql .= "FROM_UNIXTIME(" . $data . ")";

  • Character Strings and BLOBs: If BLOBs or special characters appear in a character string, then there are frequently problems with quotation marks. SQL requires that the characters ', ",\, and the zero byte be prefixed by a backslash.

    The usual way of quoting character strings that may contain special characters is via the PHP function addslashes. This function replaces the character with code 0 by \0 and prefixes the characters ', ", \, with a backslash. Please do not forget to place the result inside single quotation marks, so that SQL realizes that it is dealing with a character string:

    $sql .= "'" . addslashes($data) . "'";

    Instead of addslashes, you can also use, since PHP 4.0.3, the function mysql_escape_string, and, since PHP 4.3, the function mysql_real_escape_string:

    • mysql_escape_string functions like addslashes, but it also replaces the characters carriage return, line feed, and Ctrl Z with the strings \n, \r, and \z. The SQL syntax of MySQL does not require these additional replacements, but the MySQL logging files are thereby made easier to read.

    • mysql_real_escape_string functions like mysql_escape_string, but it also takes into account the character set of the MySQL connection. Neither the MySQL documentation (on the like-named C-API function) nor the PHP documentation gives any information on what influence the character set actually has. I have tested the MySQL server with the usa7 character set. The special characters used in German, though no longer representable, remained unchanged by mysql_real_escape_string.

    An additional alternative is to transmit such data in hexadecimal form. PHP offers the convenient function bin2hex, which transforms character strings of arbitrary length into hexadecimal. The hexadecimal character string must not be placed within single quotes.

    $sql .= "0x" . bin2hex($data);

    There is a drawback to this, however: For each byte of data, two bytes are transmitted to MySQL by PHP, thus almost double the number of bytes that are absolutely necessary.

    Remark

    If the data to be stored come from an HTML form and the PHP configuration variable magic_quotes_gpc is set to on (which is usual), then the data are already quoted. In this case there should be no additional quotation. Further information on magic quotes appears later in this chapter.

  • NULL: If data contains no data (isset($data)==FALSE) and this condition is to be stored in the database as NULL, then the following is recommended:

    $sql .= isset($data) ? "'$data'" : "NULL";

    Please note that within the SQL character command you do not set NULL in single quotes as if it were a character string.

Reading Data

The starting point of this section is again the variable data, which contains a data field that results from a query. For example, data could be filled as follows:

$result = mysql_query("SELECT ... ");
row = mysql_fetch_row($result);
$data = $row[0];

Timestamps: The variable data contains timestamps as a character string of the form 20001231235959. In PHP you cannot get very far with this. If you wish to work further with a timestamp value in PHP, you should employ the MySQL function UNIX_TIMESTAMP and formulate the SELECT query accordingly. Then data contains (for the above data) the value 978303599:

SELECT ... , UNIX_TIMESTAMP(a_timestamp) FROM ...

Date: In a DATETIME column, data contains a character string of the form 2000-12-31 23:59:59. In a DATE column, the time specification is lacking. In many instances it is most practical for the further processing of dates in PHP to use the function UNIX_TIMESTAMP here as well. You can achieve a desired format with the MySQL function DATE_FORMAT. With the following instruction, data receives a character string of the form 'December 31 2003':

SELECT ... , DATE_FORMAT(a_date, '%M %d %Y')

Time: In a TIME variable, data contains a character string of the form 23:59:59. If you do not wish to extract hours, minutes, and seconds from this character string, then you might use the MySQL function TIME_TO_SEC, which returns the number of seconds since 00:00:00. You can also use any of a number of additional MySQL functions for processing times. Caution: UNIX_TIMESTAMP does not work for TIME columns.

Pointer

You will find an overview of the functions provided by MySQL for processing dates and times in Chapter 18.

Character Strings and BLOBs: PHP fortunately has (in contrast to C) no problems with truly binary data. That is, even 0-bytes within a character string are readily processed. Thus data truly contains all the data from the database in one-to-one correspondence.

Caution

Do not forget that in general, you cannot output character strings directly with echo into an HTML document. If the character string contains special characters like <, >, ', ", or characters outside of the 7-bit ASCII character set, you must replace these characters by the corresponding HTML code via the function htmlentities.

NULL: When a data field contains NULL, then isset($data) returns the value FALSE. Similar tests could be carried out, depending on the data type, with is_numeric, is_string, etc. Please note, however, that tests on character strings with the function empty do not lead to correct results: With NULL as well as with an empty character string, empty returns the result TRUE.

Determining the Elements of an ENUM/SET

Working with ENUMs and SETs causes no difficulties: The values are passed in both directions as simple character strings. Note that with ENUMs, no additional spaces are allowed between comma-separated character strings.

Often, one wishes to display in an HTML input form all character strings of an ENUM or SET from which a selection is to be made (in a listbox, for example). To do this, you must determine with the SQL command DESCRIBE tablename columnname the definition of this field:

DESCRIBE test_enum a_enum
 Field  Type                      Null  Key  . . .
---------------------------------------------------
 a_enum enum('a','b','c','d','e') YES        . . .

The column Type of the result table contains the required information. In the following code lines the character strings enum('a','b','c') and set('a','b','c') will be abbreviated via ereg to 'a','b','c'. The function str_replace removes the single quotation marks, and then explode forms a,b,c into an array, which is stored in the variable fieldvalues. Then the while loop displays the values in the HTML document. You can use similar code to create an HTML form with radio buttons or a listbox using SET or ENUM:

$result = mysql_query("DESCRIBE test_enum a_set");
$row = mysql_fetch_object($result);
$fieldtype = $row->Type; // enum(  . . .    ) or  set(  . . .    )
ereg("\((.*)\)", $row->Type, $tmp);     // xyz('a','b','c')
                                       // --> 'a','b','c'
$tmp = str_replace("'", "", $tmp[1]); // 'a','b','c' --> a,b,c
$fieldvalues = explode(",", $tmp);
// a, b, c display in rows
while($i = each($fieldvalues)) {
echo "<br />", htmlentities($i[1]), "\n";
}
Pointer

The code directly above assumes that no commas appear within the SET and ENUM character strings. If there are commas, then the decomposition (pedagogically less intuitive, but syntactically more stable) can be expressed as follows:

ereg("('(.*)')", $row->Type, $tmp);
$fieldvalues = explode("','", $tmp[2]);

Processing Form Data (Magic Quotes)

This is not a book on PHP. However, the correct processing of form data and the storage of such data in a MySQL database leads so often to problems that it seems a good idea to discuss here some HTML and PHP basics.

Forms for Beginners

There are various ways of representing forms in HTML code. In the examples of this book, forms are generally sent with POST, and old tags (which almost every browser understands) are used for the representation of buttons. Then the construction of a simple form that consists of two text fields (type="text") and an OK button type="submit") looks like this:

<form method="POST" action="process.php">
  <p><input type="text" name="name1" size="30" maxlength="30" /></p>
  <p><input type="text" name="name2" size="30" maxlength="30" /></p>
  <p><input type="submit" name="submitbutton" value="OK" /></p>
</form>
Pointer

Further information on the construction of forms is given in every HTML book.

Evaluation of Forms

When the user clicks the OK button, the page process.php is called (action attribute of the form tag). In the PHP code you can output, since PHP 4.1, the content of both text fields with $_POST['name1'] and $_POST['name2']. To avoid error messages, you should check with if(array_key_exists('name1', $_POST)) whether the field $_POST has an entry with this key. This task is taken over in my examples by the function array_item:

function  array_item($ar, $key) {
    if(array_key_exists($key, $ar)) return($ar[$key]); }
// copy form data in the variables
$submitbutton = array_item($_POST, 'submitbutton');
$name1 = array_item($_POST, 'name1');
$name2 = array_item($_POST, 'name2');
// display form data, process, etc.
echo "<p>name1 =   ", htmlentities($name1), "</p>\n";
echo "<p>name2 = ", htmlentities($name2), "</p>\n";

With PHP 4.0 and earlier versions you can do without these instructions; the form data are available at once in the variables $name1, $name2, and $submitbutton. (This has been shown to be insecure, for which reason this has not been the default since PHP 4.2.)

Note that the representation of the form and the processing of the form data can take place from the same PHP page. Simply test at the start of the PHP code whether form data were passed. If that is the case, then begin to evaluate the data; otherwise, display the empty form. (To determine for the above form whether data were passed, simply test the content of $submitbutton. If the variable contains "OK", then the PHP page was called for the evaluation of form data.)

if($submitbutton == "OK") {
  // evaluate form data
}
else {
  // display empty form
}

Placing Form Data in the Form

In practice, a PHP page often has a third task in addition to the representation and evaluation of forms, namely, the checking whether the input is error-free. If that is not the case, then the form together with an error message is displayed, where existing form data are replaced (so that the user does not have to input everything anew).

The code for inserting already known form data into a form usually looks like the following. (The PHP code within the HTML code appears in boldface.)

<form method="POST" action="magictest.php">
<p><input type="text" name="name1" size="30" maxlength="30"
    value="<?php echo $name1; ?>"/></p>
<p><input type="text" name="name2" size="30" maxlength="30"
    value="<?php echo $name2; ?>"/></p>
<p><input type="submit" name="submitbutton" value="OK" /></p>
</form>

By this step it is seen that working with forms becomes complicated:

  • Because of magic quotes, it can happen that $name1 does not contain exactly the same text as what the user originally provided. (More later on magic quotes.)

  • You must consider what happens if the user inputs special characters in the form such as quotation marks or HTML tags.

Let us assume that in the first text field of the form the text Hotel "Golden Coyote" is input. Then $name1 contains this character string. (We will not consider magic quotes just yet in order not to complicate the issue.) If you simply place the content of $name1 in the form, then the following HTML code results, which the web browser sees:

<p><input type="text" name="name1" size="30" maxlength="30"
    value="hotel "Golden Coyote""  /></p>

It is not difficult to see where the syntax error is located. What the web browser makes of it depends on its implementation, but as a rule, in the first input field only Hotel is displayed, and the rest simply disappears. It should be clear that (perhaps with evil intent) HTML tags input into the text field can make the situation even more complex.

The usual solution to this problem consists in changing the content of form variables before replacing it in the form using htmlspecialchars. This function replaces the special characters ", &, <, and > as follows:

" &quot;

& &amp;

< &gt;

&lt; >

The code for representing the form looks like this:

<form method="POST" action="magictest.php">
  <p><input type="text" name="name1" size="30" maxlength="30"
     value="<?php echo htmlspecialchars($name1); ?>"  /></p>
  <p><input type="text" name="name2" size="30" maxlength="30"
     value="<?<?php echo htmlspecialchars($name2); ?>"  /></p>
  <p><input type="submit" name="submitbutton" value="OK" /></p>
</form>

Magic Quotes

The significance of magic quotes is that the PHP interpreter alters character strings that are read in from an external data source.

The special characters ', ", \, and the 0-byte are replaced by \', \", \\, and \0, respectively. (This operation can be carried out manually with the PHP function addslashes.)

In general, magic quotes have to do with data that come from outside, for example, from GET/POST forms, cookies, and SQL queries. In many cases this automatic alteration simplifies the further processing of character strings, but often it leads to confusion and chaos.

PHPrecognizes three magic quote modes, which are set in php.ini.

  • magic_quotes_gpc: GPC stands for get/post/cookie. That is, when a variable is set, data from these three sources are changed. Note that this affects all data in a form that are processed in PHP scripts.

  • magic_quotes_run-time: When this variable is set, PHP in most cases quotes those character strings that come from external data sources (for example, from MySQL).

  • magic_quotes_sybase: The character ' is quoted by doubling it (that is, by '' instead of by \'). However, this holds only when one of the other two magic_quote configuration variables is set.

Magic Quote Default Setting

In the default setting of PHP 4.0 to 4.3, magic_quotes_gpc is set to On, while the two other modes are Off. You will find this setting at most ISPs. It has the following consequences for programming:

  • Data transmitted from forms can be used in INSERT commands without alteration. The critical characters ', ", and \ are already quoted:

    $sql = "INSERT ... VALUES('$formVar')";

  • If you compare data in a form with data from an SQL query, you must first free the form data from backslash characters with the PHP function stripslashes( ):

    if(stripslashes($formVar) == $sql_result_row[0]) ...

  • On the other hand, if you make the comparison not in PHP, but with MySQL (as part of a WHERE conditional), then no extra measures are required:

    $sql = "SELECT ... WHERE authorsName = '$formVar'";

  • If you wish to display the form data in an HTML document (or in a new form, say to allow corrections to be made), then you must first delete the quotation marks with stripslashes and then quote anew (and differently!) all HTML-specific special characters with htmlentities:

    echo htmlentities(stripslashes($formVariable));
    Warning

    In php.ini.recommended (see Chapter 2), all three modes are switched to Off for reasons of speed optimization.

    Before you begin developing your own code, you should certainly determine which magic quote settings hold on your ISP's server.

Determining and Changing Magic Quote Modes in a PHP Script

To some extent, you can determine or change the three configuration variables in the code of your PHP script:

  • magic_quotes_gpc: get_magic_quotes_gpc( ) determines the current status. A change cannot be made.

  • magic_quotes_run-time: get_magic_quotes_runtime( ) determines the current status; set_magic_quotes_runtime( ) changes it.

  • magic_quotes_sybase: The status can be neither determined nor altered.

Removing Magic Quotes

Without doubt, magic quotes were created with the intention of simplifying the lives of programmers. In practice, alas, the opposite is often the case. When you develop code in which a large amount of data is read from forms and is later replaced in the same or another form, the automatic quoting of special characters often causes more problems than it solves.

If you are working with PHP 4.1 or higher, you can turn off the quoting by calling the following function:

function  no_magic() {   // PHP >= 4.1
    if (get_magic_quotes_gpc()) {
      foreach($_GET as $k=>$v)    $_GET["$k"] = stripslashes($v);
      foreach($_POST as $k=>$v)   $_POST["$k"] = stripslashes($v);
      foreach($_COOKIE as $k=>$v) $_COOKIE["$k"] = stripslashes($v);
    }
}

Through and including PHP 4.0 you can instead use the following code (the code cannot, however, be placed in a function, since the fields $HTTP_xxx_VARS are not superglobal):

if (get_magic_quotes_gpc()) {   // PHP < 4.1
    foreach($HTTP_GET_VARS as $k=>$v)
      $HTTP_GET_VARS[$k] = stripslashes($v);
    foreach($HTTP_POST_VARS as $k=>$v)
      $HTTP_POST_VARS[$k] = stripslashes($v);
    foreach($HTTP_COOKIE_VARS as $k=>$v)
      $HTTP_COOKIE_VARS[$k] = stripslashes($v);
}

Then, you must read out the form variables with $HTTP_POST_VARS['name'].

Experimenting with Forms and Magic Quotes

If you do not yet have much experience with forms or magic quotes, I would heartily recommend to you to experiment a bit with them. The following PHP script testmagic.php consists of a simple form allowing for input into two text fields (name1 and name2). If you click on OK, then the form data are passed to the same file, displayed with and without magic quotes, and then replaced in the form fields, so that they can be further edited.

Now try to input special characters like ä, é, ü, ô, \, ', <, >, or entire HTML tags. There should be no problems; that is, the special characters are displayed correctly. Now see what happens if you do without the functions no_magic and/or htmlentities (see Figure 11-3). Look at the HTML source code. (Most web browsers offer this possibility, Mozilla, for example, with Ctrl+U.)

Click To expand
Figure 11-3: Testing magic quotes
<!-- file general/magictest.php        -->
<html><head>
  <meta http-equiv="Content-Type"
        content="text/html; charset=iso-8859-1" />
  <title>PHP Programming, magic test</title>
</head><body>
<?php {

  function  array_item($ar, $key) {
   if(array_key_exists($key, $ar))
     return($ar[$key]); }
  function  no_magic() {   // needs PHP >= 4.1
   if (get_magic_quotes_gpc()) {
    foreach($_GET as $k=>$v)       $_GET["$k"] = stripslashes($v);
    foreach($_POST as $k=>$v)     $_POST["$k"] = stripslashes($v);
    foreach($_COOKIE as $k=>$v) $_COOKIE["$k"] = stripslashes($v);
   }
  }
// magic_quotes_gpc show statius
echo "<p>get_magic_quotes_gpc = ", get_magic_quotes_gpc(), "</p>\n";
    // copy form input into variables
    $submitbutton = array_item($_POST, 'submitbutton');
    $name1 = array_item($_POST, 'name1');
    $name2 = array_item($_POST, 'name2');
    // Is there any form input at all?
    if($submitbutton == "OK") {
      // disply input with magic quotes
      echo "<h2>Before no_magic()</h2>\n";
      echo "<p>name1 = ", $name1, "\n";
      echo "<br />name2 = ", $name2, "</p>\n";
      echo "<p>htmlentities(name1) = ", htmlentities($name1), "\n";
      echo "<br />htmlentities(name2) =   ", htmlentities($name2),
        "</p>\n\n";
      // display input without magic quotes
      no_magic();
      echo "<h2>After no_magic()</h2>\n";
      $name1 = array_item($_POST, 'name1');
      $name2 = array_item($_POST, 'name2');
      echo "<p>name1 = ", $name1, "\n";
      echo "<br />name2 = ", $name2, "</p>\n";
      echo "<p>htmlentities(name1) = ", htmlentities($name1), "\n";
      echo "<br />htmlentities(name2) =   ", htmlentities($name2), "</p>\n";
    }
} ?>
<h2>Test form</h2>
<form method="POST" action="magictest.php">
<p><input type="text" name="name1" size="30" maxlength="30"
    value="<?php echo htmlspecialchars($name1); ?>" /></p>
<p><input type="text" name="name2" size="30" maxlength="30"
    value="<?php echo htmlspecialchars($name2); ?>" /></p>
<p><input type="submit" name="submitbutton" value="OK" /></p>
</form>
</body></html>

Summary

In processing form data, you should take the following into consideration:

  • Before further processing of form data, turn off, if necessary, magic quotes. The additional backspace characters will otherwise interfere with further processing.

  • When you replace form input back into a form, use the function htmlspecialchars. You thereby avoid the invalid HTML code caused by the characters ", &, <, >.

  • If you wish to represent form data in an HTML document (not within the input field of the form), then use the function htmlentities.With this, you achieve that all special characters (not only ", &, <, and <) are represented by the correct HTML code.

  • If you wish to store form data in a MySQL database, then use the function addslashes (or mysql_escape_string or mysql_real_escape_string). You thereby prevent any input single quotes, double quotes, or backspaces from causing syntax errors in the SQL character strings.


Error Checking

All mysql_xxx functions return FALSE if an error occurs during execution. You obtain information as to the type of error by evaluating the functions mysql_errno (returns the error number) and mysql_error (error message):

$result=mysql_query($sql);
if(!$result) {
    echo "<P>error: ", mysql_errno(),
    ": " . htmlentities(mysql_error()), ".</p>\n";
}

One always must reckon with the reality of errors, and in particular in the execution of SQL commands that alter data. One possible source of error is in the connection to the MySQL server.

In the development of PHP scripts the function mysql_query_test is often very useful in searching for the cause of the error. It is meant as a replacement for mysql_query, displaying every executed SQL command in blue. If an error occurs, then an error message will be displayed as well:

// example file general/mylibraryconnect.inc.php
function  mysql_query_test($sql) {
  echo '<p><font color="#0000ff"> SQL:', htmlentities($sql),
   "</font></p>\n";
  $result = mysql_query($sql);
  if($result) return($result);
  echo '<p><font color="#ff0000">Error: ',
     htmlentities(mysql_error()),
  "</font></p>\n";
  die();
}
Pointer

A list of all error messages can be found in the files errmsg.h and mysqld_error.h.Bothfilesarepartofthesource-code package for MySQL.

Error Search

I am constantly receiving e-mail, and there appear contributions in discussion lists, about my book (see http://www.kofler.cc) with the question, What is one to do when an example from the book does not function properly? The error message (if indeed there is one) often gives no clear idea as to the cause of the problem, instead giving a cryptic message like Supplied argument is not a valid MySQL result resource. This means that you are attempting to evaluate a query result (in the examples of this book generally in the variable $result) that contains no data due to the error that has occurred.

The cause of the problem is almost always access to the database: The PHP script is unable or is forbidden to read the data from the database and therefore returns an incorrect result or no result at all. There are many reasons why this can happen, but it almost always has to do with access privileges: incorrect password, no password (although one was required). This topic is dealt with extensively in Chapter 9. Only when MySQL is completely unsecured does the combination name = "root" and password = "" work with accesses by the local computer.

The best remedy for determining the cause of the error is the function mysql_query_test introduced above. Simply replace all mysql_query functions with mysql_query_test( ) and test the page again. Then you will at least know what is not working.


Storing and Displaying Images

A frequently posed question in PHP and MySQL newsgroups and discussion forums is this: How can I give visitors to my web site a way to upload their own images and then display them? This section answers this question with an example project (see Figure 11-4) that demonstrates the required programming techniques.

Click To expand
Figure 11-4: Storing and displaying images

Fundamentals and Programming Techniques

Database Design

The starting point for our example is the database test_images with the table images. This database and the table can be created with the following commands:

CREATE DATABASE  test_images
USE test_images
CREATE TABLE  images (
  id    BIGINT NOT NULL AUTO_INCREMENT,
  name  VARCHAR(100) NOT NULL,
  type  VARCHAR(100) NOT NULL,
  image MEDIUMBLOB NOT NULL,
  ts    TIMESTAMP(14) NOT NULL,
  PRIMARY KEY (id))

The images table contains, in addition to the usual primary index id, the column name to describe the image, type to store the image format (e.g., "JPEG IMAGE"), image to store the image data (where images have maximum size 16 megabytes), and ts to log the time of saving.

Warning

From the standpoint of efficiency, it is not a good idea to store images or other large binary files in a database. For that, a traditional file system is more suitable. Databases with many BLOBs suffer in the area of speed above all because it is difficult for the database system to store frequently needed data and indexes in RAM. It would therefore make more sense to store images as ordinary files and enter only the file name in the database.

In practice, however, the storage of binary data in BLOBs occurs frequently, and for two reasons: First, it is easier to deal with data (e.g., backup, server migration) if all data are stored in a central location, and second, many ISPs will not allow you to create and then read out local files in PHP scripts or other code.

HTML Form for File Transfer

If you wish to offer the possibility on an HTML page for file transfer to the web server, then you will require a form with a particular structure. It is crucial that the attributes method and enctype be correctly set. The input field for the file name (to which the browser automatically adds the button BROWSE) is formed with an <input> tag with the attribute type = "file". If you wish to place a limit on the file size, you can do that with a hidden field with the attribute name = "MAX_FILE_SIZE". The following lines show the code for the upload form in Figure 11-4, where the file size is limited to 100 kilobytes:

<form method="post" action="pictures.php"
      enctype="multipart/form-data">
  <p>Description: <input name="descr" type="text" /></p>
  <p>Image file: <input name="imgfile" type="file" /></p>
  <p><input type="submit" value="OK" name="submitbtn" /></p>
  <input type="hidden" value="102400" name="MAX_FILE_SIZE" />
</form>

PHP Evaluation of the $_FILES Field

The PHP script that is called with the form data contains the usual form fields as usual in the $_POST field (that is, $_POST['descr']) with a description of the image as well as $_POST['submitbtn'] with the character string "OK". The information on the transferred file is found in the $_FILES field. Access follows according to the above example form via the key 'imgfile' (see the name attribute for <input type="file">) and again returns a field. Its elements give information about the transferred file:

$imgfile = $_FILES['imgfile'];
$name    = $imgfile['name'];    // file name (without drive/path)
$type    = $imgfile['type'];    // file type, e.g., "image/gif"
$size    = $imgfile['size'];    // file size in bytes
$uperr   = $imgfile['error'];   // error number (0 = no error)
$tmpfile = $imgfile['tmp_name']; // name of the local temporary file
Remark

$imgfile['error'] has been available only since PHP 4.2, and $_FILES only since PHP 4.1. With older versions of PHP (4.0 and earlier), you can read the information directly from the variables $imgfile (local file name, corresponds to $tmpfile), $imgfile_name, $imgfile_type, and $imgfile_size.

The transferred file is stored temporarily by the web server in a temporary file (file name $tmpfile). This file can be read by the PHP script, for example, with the functions fopen and fread. Note that the file is automatically deleted after the execution of the PHP script.

From the point of view of security, file transfer is a dangerous operation. Therefore, it is recommended to secure the code for further processing of the file. The following query tests whether $tmpfile is empty, whether a transfer error has occurred, or whether the given file is not a transferred file (is_uploaded_file) but a local file. This security is important, so that an attacker cannot attempt to evaluate a local system file (e.g., the Unix/Linux password file /ect/passwd). Such cases are known to have occurred:

if(!$tmpfile or $uperr or !is_uploaded_file($tmpfile))
  echo "<p>error  . . .    </p>\n";
else {
     . . .    // read and process the temporary file
}

Optionally, you can also evaluate the file type and size in the security query.

Storing a Transferred File in the MySQL Database

To store the transferred file in the MySQL database, read the entire file into a variable and execute an INSERT command. In the simplest case, the code looks like this:

$file = fopen($tmpfile, "rb");   // file open (read-only, binary)
$imgdata = fread($file, $size); // read file
fclose($file);
mysql_query("INSERT INTO images (image) " .
      "VALUES ('" . addslashes($imgdata) . "')");

In the code of our example program, the INSERT command is somewhat more complex, since in addition to the MIME type of the image (e.g., "GIF image"), its name or a description will be stored.

Displaying Images from the MySQL Database

It is somewhat more complicated to read an image out of the MySQL database and display it within an HTML document. It is fundamentally impossible within a single PHP script to create an HTML document (which is the usual task of a PHP script) and at the same time to generate the data for the representation of an image. The two tasks must be divided between two PHP scripts.

The first (usual) script creates the HTML page, including the <img> tag for the image. In this tag, however, instead of the usual name of a local image file, the name of the second PHP script is specified together with the ID number of the image:

// PHP-Script pictures.php
echo "<img src=\"showpic.php?id=$id"\" />";

The browser sees, for example, the tag <img src="showpic.php?id=3" />, and now calls the second script showpic.php?id=3. Its code, somewhat simplified, looks like this:

// PHP-Script showpic.php
$result = @mysql_query(
  "SELECT image FROM images WHERE id = $id");
$row = @mysql_fetch_object($result);
header("GIF image");   // file type
echo $row->image;

In fact, a bit more code is required to create the database connection, to determine $id, and secure the script against possible errors. If the image database contains not only GIF images, but also bitmaps in other formats, then the file type set with header must also be retrieved from the database. At this place it is crucial that the second PHP script not generate an HTML file and in general, not perform any tasks with echo before the header function is called. This function instructs the PHP interpreter to create not, as usual, an HTML file, but an image file.

Processing Large Files

If you wish to allow the transfer of very large files, you must note a few things:

  • The limit for the maximal size of files to be transferred must be set sufficiently high. The limit is set in php.ini with the variable upload_max_file_size (default value 2 megabytes).

  • The limit for the POST transfer must also be high enough (php.ini, variable post_max_size, default value 8 megabytes).

  • The PHP interpreter must be capable of reserving enough memory to process files of the maximal size that has been set (php.ini, variable memory_limit, default value 8 megabytes).

  • The PHP interpreter must give the script sufficient time for processing the data (php.ini, variable max_execution_time). The default value of 30 seconds is generally too small for large files.

  • You may also need to increase the variable max_input_time (default value 60 seconds). This variable specifies how long a wait will be permitted for transfer of data over the Internet.

  • For the transfer of the image between the PHP script and the MySQL server to succeed, the maximal packet size (which, for example, the INSERT command limits) must be sufficiently large. The MySQL variable max_allowed_packet can be changed either in my.cnf or with the SQL command SET max_allowed_packet=n. The default value is only one megabyte.

Program Code

The entire code of the example project is contained in three files, which are located with the PHP example files in the directory pictures:

connect.inc.php | Makes available the functions connect_to_picdb and array_item.
pictures.php | Displays a table with the most recent ten images and a form for the transfer of additional images. The script contains as well the code for form evaluation.
showpic.php | Displays a single image.

The File connect.inc.php

This file contains two functions: connect_to_picdb creates a connection to the MySQL database test_pictures, while array_item reads an element from a PHP field (if such exists):

<?php       //   -*- C++ -*-
// file pictures/connect.inc.php
function  connect_to_picdb() {
  $mysqluser="root"; // username
  $mysqlpasswd="xxx";         // password
  $mysqlhost="localhost"; // name of computer MySQL is running
  $connID = @mysql_connect($mysqlhost, $mysqluser, $mysqlpasswd);
  if ($connID) {
     mysql_select_db("test_images");   // default database
     return $connID;
  }
  else {
    echo "  . . .    error  . . .    ";
    exit();
  }
}
function  array_item($ar, $key) {
  if(array_key_exists($key, $ar))
     return($ar[$key]); }
?>

The File pictures.php

The code in pictures.php consists of three parts. The first part is responsible for the processing of form data. If a file is transferred without error, then its content is stored in the table images.

The switch construction evaluates the type of transferred file and from this makes a new MIME character string. (MIME stands for Multipurpose Internet Mail Extensions. Originally, MIME was used only for e-mail attachments, but in the meanwhile, MIME has come to control how a web browser should deal with various file types.) The MIME type must be specified when an image is displayed and is therefore stored together with each image in the images table. Unfortunately, the type information does not correspond to MIME, but rather uses different character strings (which on top of everything, depend on the web browser):

  <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN">
  <?php {   // file pictures/pictures.php
    include("connect.inc.php");
    $connID = connect_to_picdb();
  } ?>
  <html><head>
  <meta http-equiv="Content-Type"
    content="text/html; charset=iso-8859-1" />
  <title>PHP Programming, image upload</title>
  </head><body>
  <?php {
    //  part 1: form evaluation, store image in MySQL database
    $submitbtn = array_item($_POST, 'submitbtn');
    $descr =         array_item($_POST, 'descr');
    $imgfile =     array_item($_FILES, 'imgfile');
    // is there form data to process?
    if($submitbtn == 'OK' and is_array($imgfile)) {
        $name       = $imgfile['name'];
        $type       = $imgfile['type'];
        $size       = $imgfile['size'];
        $uperr     = array_item($imgfile, 'error'); // PHP >= 4.2.0
        $tmpfile = $imgfile['tmp_name'];
        if(!$descr) $descr = $name;
        switch ($type) {
        case "image/gif":
    $mime = "GIF Image";   break;
        case "image/jpeg":
        case "image/pjpeg":
    $mime = "JPEG Image"; break;
        case "image/png":
        case "image/x-png":
    $mime = "PNG Image";   break;
        default:
    $mime = "unknown";
        }
        // security test
        if(!$tmpfile or $uperr or $mime == "unknown" or
              !is_uploaded_file($tmpfile))
    echo "<p>An error occured when processing the form data:
              Perhaps you forgot to specify an
              image file or the file is too large
              or the image type is unknown.</p>\n";
        else {
    // read the transferred file and store it in the database
    $file = fopen($tmpfile, "rb");
    $imgdata = fread($file, $size);
    fclose($file);
    mysql_query(
              "INSERT INTO images (name, type, image) " .
              "VALUES ('" . addslashes($descr) . "', " .
              "    '$mime', " .
              "     '" .  addslashes($imgdata) . "')");

        }
  }

The second code segment in pictures.php serves to read the last ten images from the database and display them in a table. As we have already described, the actual display of the images is delegated to the script showpic.php:

    //  part 2: display the last ten images as a table
    echo "<h2>Images recently uploaded  . . .    </h2>\n";
    $result = mysql_query(
        "SELECT id, name, " .
        "DATE_FORMAT(ts, '%Y/%c/%e %k:%i') " .
        "FROM images ORDER BY ts DESC LIMIT 10");
    $rows = mysql_num_rows($result);
    $cols = mysql_num_fields($result);
    if($rows==0)
        echo "<p>No images in database  . . .    </p>\n";
    else {
        echo "<table border=1>\n<tr>";
        for($i=0; $i < $rows; $i++)   // images
    echo "<th valign=\"bottom\"><img src=\"showpic.php?id=" .
          mysql_result($result, $i, 0) . "\" /></th>";
        echo "</tr>\n<tr>";
        for($i=0; $i < $rows; $i++)   // imagename/description
    echo "<th>" . htmlentities(mysql_result($result, $i, 1)) .
          "</th>";
        echo "</tr>\n<tr>";
        for($i=0; $i < $rows; $i++)   // date and time
    echo "<th>" . mysql_result($result, $i, 2) . "</th>";
        echo "</tr>\n</table>\n";
    }
} ?>

If the images are to be displayed underneath one another instead of side by side, then the code for generating the table can be simplified a bit:

echo "<table border=1>\n";
while($row = mysql_fetch_object($result)) {
    echo "<tr>";
    echo "<th>" . htmlentities($row->name) . "</th>";
    echo "<th>$row->ts</th>";
    echo "<th><img src=\"showpic.php?id=$row->id\" /></th>";
    echo "</tr>\n";
}
echo "</table>\n";

The third part of pictures.php contains the HTML code for the form for transferring additional pictures:

<h2>Upload your picture  . . .    </h2>
<p>Maximum size 100 kByte, only PNG, JPEG and GIF formats supported.</p>
<form method="post" action="pictures.php" enctype="multipart/form-data">
  <input type="hidden" value="10240000" name="MAX_FILE_SIZE" />
  <p>Description: <input name="descr" type="text" /></p>
  <p>Image file:   <input name="imgfile" type="file" /></p>
  <p><input type="submit" value="OK" name="submitbtn" /></p>
</form>
</body></html>

The Script showpic.php

The script showpic.php displays a single image, whose ID number is passed as parameter in the address. The code has already been described, and is here simply extended with a few security queries:

<?php {   // file  pictures/showpic.php?id=n
  include("connect.inc.php");
  // evaluate id parameter; exit, if no id
  $id = array_item($_GET, 'id');
  if(!$id) exit;
  // load image from the database
  $connID = connect_to_picdb();
  $result = @mysql_query(
    "SELECT image, type FROM images WHERE id = $id");
  if(!$result) exit;
  // display the image
  $row = @mysql_fetch_object($result);
  if(!$row) exit;
  header($row->type);
  echo $row->image;
} ?>
Tip

If an error occurs in showpic.php, it is difficult to locate, since showpic.php is not normally displayed as an independent HTML document (and therefore, no error messages are displayed). However, for error searching you can try out showpic.php independently of pictures.php by specifying in your web browser the address http://servername/pictures/showpic.php?id=123,for example.


Chapter 12: PHP - Library Management

Team LiB

Chapter 12: PHP — Library Management

This chapter describes a relatively large PHP example program that allows for querying the mylibrary database and adding new titles to it. After the relatively brief examples of the previous chapter, which demonstrated the most important programming techniques, the mylibrary project shows the complexity of managing PHPprojects in practice.

Introduction

The example of this section is based on the database mylibrary, which was introduced in Chapter 5. It shows how a few PHP pages can be used to input book information into a database and then to search the database. Our example consists of the following files:

mylibrarypassword.inc.php | MySQL logging information ($mysqluser, -password, -host, -dbname)
mylibraryconnect.inc.php | various auxiliary functions (including database login)
find.php | search for authors/titles
simpleinput.php | input of new book titles
input.php | convienient input of new book titles
categories.php | management of the hierarchical category list

The script files are independent of one another. Within each script the transmittal of session information is carried out as necessary via URL variables (that is, name.php?variable=xxx) or by formula fields (URL stands for Uniform Resource Locator and denotes addresses like http://www.company.com/page.html).

Remark

The examples of this chapter assume at least PHP 4.1. Versions that will run under PHP 4.0 can be found at my web site http://www.kofler.cc in the PHP example files for the first edition of this book.


Auxiliary Functions (mylibraryconnect.inc.php)

The file mylibraryconnect.inc.php contains several auxiliary functions that in part have already been introduced in the previous chapter. The following list gives a brief description:

  • connect_to_mylibrary( ) creates the connection to MySQL. If the connection cannot be made, an error message is displayed. Before the function is called, no HTML code is to be generated, since the error message represents a complete HTML page (including the HTML header).

  • The function array_item($ar, $key) returns the element $ar[$key], assuming that the field $ar contains an element with the key $key.

  • build_href($url, $query, $name) creates an HTML link of the form <a href="$url?$query">htmlentities($name)</a>.

  • mysql_query_test($sql) displays an SQL command, executes it, and displays an error message if necessary. This function returns the query result, and thus can be implemented like mysql_query, and moreover, it greatly simplifies debugging.

The following lines show the most important parts of mylibraryconnect.inc.php:

<?php       // mylibrary/mylibraryconnect.inc.php
// creates the connection to the database
function connect_to_mylibrary() {
    // definiert $mysqluser, -password, -host and -dbname
    include("mylibrarypassword.inc.php");
    $connID = @mysql_connect($mysqlhost, $mysqluser, $mysqlpasswd);
    if ($connID) {
        mysql_select_db($mysqldbname);   // default database
        return $connID; }
    else {
              echo "  . . .    error message . . . ";
        exit(); }
}
// reads an element from a PHP field
      function  array_item    ($ar, $key) {
    if(array_key_exists($key, $ar))
        return($ar[$key]); }
// returns <a href=$url?$query>$name</a>
function  build_href($url, $query, $name) {
    if($query)
        return "<a href=\"$url?" . $query . "\">" .
          htmlentities($name) . "</a>";
    else
        return "<a href=\"$url\">" . htmlentities($name) . "</a>";
}
?>

Securing the mylibrary Project

The script mylibraryconnect.inc.php loads via include the file mylibrarypassword.inc.php, whose few lines of code define the four variables $mysqluser, -password, -host, -dbname:

<?php       // file mylibrary/mylibrarypassword.inc.php
  $mysqluser     = "root";     // MySQL user name
  $mysqlpasswd = "xxx";        // passwork
  $mysqlhost     = "localhost";  // computer name for the MySQL server
  $mysqldbname = "mylibrary";   // database name
?>

When you install the mylibrary project, you must first adapt the login and password information to your MySQL installation. If you wish to provide better security for the project, it is recommended to move mylibrarypassword.inc.php into another directory, one that is secured with .htaccess and whose files are therefore not accessible over the Internet. Of course, you must then correspondingly alter the include instruction in mylibraryconnect.inc.php.


Book Search (find.php)

The file find.php helps in locating authors and books in the mylibrary database. To do this, the initial letters of the title or of the author are entered into the search form (see Figure 12-1).

Click To expand
Figure 12-1: The search form

In the search results (see Figures 12-2 and 12-3), authors and publishers are executed as links, so that it is quite simple to find all the books of a given author or all books put out by a particular publisher.

Click To expand
Figure 12-2: Results of a search for book titles beginning with the letter M
Click To expand
Figure 12-3: Results of a search for authors whose last name begins with K

If search results do not fit on a page, then one can leaf page by page through the results. (The number of titles per page is set in the program by the variable $pagesize to 5. This value can easily be increased.)

Problems of Efficiency in the Representation of Search Results

The file find.php divides long lists of book titles into several pages. To this end, SELECT is called with LIMIT pagesize * (pagenr-1), pagesize. Unfortunately, there are two factors that make the paged representation of SELECT results inefficient:

  • Generally, on each page of the results, you would like to report the number of additional pages. But to do this you need to know how many records the query returns. This can be easily determined via SELECT COUNT( ... ), but if your query is complex, then the twofold execution—once with SELECT COUNT for the number and then with SELECT LIMIT for the results of the current page—is very inefficient.

    There is, alas, no optimal solution to this problem. One approach would be always to search for pagesize+1 results. If pagesize+1 results are actually found, it is then clear that there are more results (at least one, perhaps more). The program then displays pagesize results together with a MORE button, which indicates that there are additional results, but without telling how many. This is efficient, but not very elegant (find.php follows this strategy).

    Another possibility is to execute SELECT for the first page without COUNT and without LIMIT (or at least with a relatively high LIMIT value). The advantage is that with a single query you determine both the number of all data records (via mysql_num_rows) and the desired result records. The disadvantage is that if the query returns a large number of results, then all the records are transmitted from MySQL to PHP, even though relatively few records are actually to be displayed. This is probably even less efficient than two SELECT queries.

    Since MySQL 4.0 there has been a better variant: Execute the first SELECT command with LIMIT and the option SQL_CALC_FOUND_ROW. You obtain now, to be sure, only n records, but you can determine with the command SELECT FOUND_ROWS( ) how many records there are altogether.

    If you decide to determine the number of result records (by whatever means), then you should do this only for the first result page. For all further pages you should store this information in a session or transmit it as part of the link.

  • The second problem is that for each page, the entire SELECT query with somewhat altered LIMIT parameters must be executed anew. Of course, the LIMIT clause prevents all the result records from being transmitted each time from MySQL to PHP, but depending on the query type, the cost is the same at least for MySQL. (This holds especially for queries that join information from several tables with JOIN, that sort or group the results, or that contain complex WHERE conditionals that cannot be resolved directly with the help of an index.)

    Again, there is no perfect solution. Many programming languages offer the possibility of temporarily storing intermediate results, but not PHP. (Theoretically, you can store the query results in a new table, but then how and when will this table be deleted? Perhaps the user will be satisfied with the first search results anyhow, and then it would be excessively costly to construct a new table.)

    The only generally valid recommendation is this: Display as many results per page as possible and thus avoid the user having to do too much paging. (If your main goal is to achieve the maximum number of page accesses, then of course, this is not the optimal recipe. In find.php the number of titles per page is limited for a different reason to five: The example database contains relatively few titles, and with pagesize=25 you would scarcely have the chance to formulate a query that would yield that many results.)

Program Structure

The starting point for the program is the search form displayed in Figure 12-1. To evaluate the form's input we again call find.php. The search results and the page references again refer to find.php. In this case information such as the search item and the current page number are passed as part of the link (find.php?sqlType=author& ...).

In find.php, therefore, we must first analyze how the script was called: If no data are transferred, then find.php simply displays the empty form. But if there are predefined variables, then the program forms a corresponding SQL query and then displays the result of the search.

Code Outline

The following lines give an overview of the code in find.php, which should help in understanding the following extensive description:

<!-- php/mylibrary/find.php  -->
<?php
// insert auxiliary functions
include("mylibraryconnect.inc.php");
//  read form and URL variables
// (necessary only with PHP >= 4.1)
    $sqlType     = array_item($_REQUEST, 'sqlType');
    $authID      = array_item($_REQUEST, 'authID');
    $publID      = array_item($_REQUEST, 'publID');
    $search      = array_item($_REQUEST, 'search');
    $page        = array_item($_REQUEST, 'page');
    $formSearch  = array_item($_POST,       'formSearch');
$formSubmitTitle = array_item($_POST,       'formSubmitTitle');
$formSubmitAuthor = array_item($_POST,       'formSubmitAuthor');
// cancel URL encoding
$search = urldecode($search);
// slashes (eliminate magic quotes)
if(get_magic_quotes_gpc()) {
    $search = stripslashes($search);
    $formSearch = stripslashes($formSearch); }
// define additional auxiliary functions (see description
// in the next section)
function  last_name_last($x) {  . . .    }
// establish link to the database
$connID = connect_to_mylibrary();
?>
<!-- start of the HTML document -->
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN">
<html><head>
<html><head>
    <meta http-equiv="Content-Type"
    content="text/html; charset=iso-8859-1" />
<title>PHP Programming, find titles (mylibrary database)</title>
</head><body>
<h2>Search for titles in the mylibrary database</h2>
<?php
  // Main-Code
  //initialize $pagesize, validate $page
  // if called without parameters:
  // display form
  // initialize $sql, $sqlcreate, $sqllimit
  // if there are form variables:
  // form SQL query corresponding to form input (--> $sql)
  // if there are URL variables:
  // form SQL query corresponding to the URL variables(--> $sql)
  // if the variable $sql is not empty
  // execute SQL query
  // display results
  // if $page>1 or if there are additional results
  // display links to other pages with search results
?>
</body></html>
Pointer

The function last_name_last moves the first word in a character string to the end of the string. This turns the database-specific "Kennedy John F." into the more readable "John F. Kennedy". The code for this function and for the inverse function last_name_first appears in the next section.

Form Variables

  • formSearch contains the search character string (that is, the initial letters of the author or title sought).

  • formSubmitTitle contains "Search for title" if the user clicks on the button for title search.

  • formSubmitAuthor contains "Search for author" if the user clicks on the button for author search.

URL Variables

  • sqlType contains "title" if the HTTP address contains further variables for the display of search results for a title search.

  • authID contains the ID number of an author if a title by this author is to be displayed.

  • publID contains the ID Number of a publisher if a title put out by that publisher is to be displayed.

  • search contains the search character string if the title with these initial letters is to be displayed. Note that the search string in search is encoded with urlencode and must be decoded with urldecode. Moreover, the string can contain backslashes (magic quotes), which must be removed.

  • page contains the page to be displayed. (The program code ensures that values less than zero and larger than 100 cannot be passed.)

Displaying the Form

If sqlType is empty (that is, if data were not passed via the HTTP address) and if either no form data or impermissible form data are at hand (that is, an empty search string), then the form is displayed:

if(empty($sqlType) and empty($formSearch)) {
?>
<form method="post" action="find.php">
<p>Please specify the first letters of a book title or
      of an authors last name.</p>
<p>Search for:
<input name="formSearch" size="20" maxlength="20" /></p>
<p>
<input type="submit" value="Search for title"
    name="formSubmitTitle" />
<input type="submit" value="Search for author"
    name="formSubmitAuthor" /></p>
</form>
<?php
    } // end  of if (  . . .    )

Forming the SQL Command for Title or Author Search

In the following, our task is to formulate an SQL command for a search for a book title or author. Regardless of the database to which this query will be referred, several variables must be initialized.

The variable pagesize specifies how many titles can be displayed per page. If the variable page contains a value, this value specifies which page will be the next to be displayed. This variable is imported via the HTTP address and contains within it the risk of misuse. (Even if an evil-minded user specifies input.php?page=10000000000 as HTTP address, the program should react in a reasonable manner and not display links to the previous 9999999999 pages.) The permissible range for page will therefore be limited to the range 1–100.

// number of titles per page
$pagesize = 5;
// validate page variable
if($page>100)
    $page=100;
elseif($page<1)
    $page=1;
elseif(!is_numeric($page))
    unset($page);

The SELECT command for the title or author search is formed in the variable sql. If a title is the search object, then a title search takes place in five steps:

  • The database tmpTitleIDs is deleted if it already exists (for example, because the script was broken off during its previous execution due to an error and was not executed to the end).

  • With a SELECT query, the titleID numbers of the sought titles (only for the page to be displayed) are returned. These results are stored in the temporary table tmpTitleIDs.

  • In a second query, the detailed information (that is, book title, subtitle, category, etc.) for each titleID number is determined from the tables titles, categories, languages, etc.

  • In a third query, the tables titles, rel_title, and authors are consulted to determine the authors for each title.

  • After the evaluation of all these data, the temporary table tmpTitleIDs is deleted.

At first, a title query proceeds only to the second step. In sql, therefore, only the command for the generation of the tmpTitleIDs table is temporarily stored. The beginning and end of this command are the same: It is the part of the command for the generation of the temporary table (CREATE TEMPORARY ... ) and the part for limiting the search results to one page (LIMIT ... ). To avoid redundant code, these parts of the SQL command are placed in advance in the variables sqlcreate and sqllimit:

Pointer

The reason for working here with a temporary table and thereby determining title and author details separately is discussed in Chapter 5.

The temporary table is very small: It contains at most pagesize ID numbers. To optimize speed, the table type HEAP is chosen. This ensures that the table is generated exclusively in the MySQL server's RAM, not on the hard disk.

// predefine variables
$sql = "";
$sqlcreate = "CREATE TEMPORARY TABLE tmpTitleIDs TYPE=HEAP ";
if(isset($page))
  $sqllimit = "LIMIT " .  (($page-1) * $pagesize) . "," .
    ($pagesize + 1);
else
  $sqllimit = "LIMIT " .  ($pagesize + 1);

Creating SQL Queries from Form Data

If form data are available, then the following lines create the SQL command on that basis. First, the user input is separated from the PHP magic quotes. Then the special characters <, >, ", %, *, _, and \ are eliminated. The one remaining problematic special character, namely, the apostrophe, is permissible, since it can appear in names (e.g., O'Reilly):

// formulate SQL command from form data
if(isset(isset($formSearch)) {
  //remove < > " % _ * \  from $formSearch
  $search = trim(stripslashes($formSearch));
  $remove = "<>\"_%*\\";
  for($i=0; $i<strlen($remove); $i++)
     $search = str_replace(substr($remove, $i, 1), "", $search);

Depending on whether the search is for an author or a title, the program displays a brief text about the content of the search and then constructs sql according to one of the two following patterns. Note, please, that in the case of an author search a relatively high limit (not pagesize) is used. This is because the program anticipates more than one page in the search result only in the case of a title search.

SELECT authName, authID FROM authors
WHERE authName LIKE 'abc%'
ORDER BY authName LIMIT 100
CREATE TEMPORARY TABLE tmpTitleIDs TYPE=HEAP
SELECT titleID FROM titles " .
WHERE title LIKE 'abc%'
ORDER BY title LIMIT 21

Here is the PHP code for assigning this command to the variable sql:

  if($search=="")
      unset($search);
  else {
   // search for authors
   if($formSubmitAuthor) {
   echo "<p>Search for authors whose family names begin with <i>",
      htmlentities($search), "</i>:</p>\n";
   $sqlType = "author";
   $sql =
     "SELECT authName, authID FROM authors " .
     "WHERE authName LIKE '" . addslashes($search) . "%' " .
      ORDER BY authName LIMIT 100";
      }
      // search for titles
      else {
   echo "<p>Search for titles beginning with <i>",
     htmlentities($search), "</i>:\n</p>";
   $sqlType = "title";
   $sql = $sqlcreate .
    "SELECT titleID FROM titles " .
    "WHERE title LIKE '" . addslashes($search) . "%' " .
    "ORDER BY title " .  $sqllimit;
      }
    }
}

Creating SQL Queries from URL Variables

One can deduce the existence of URL variables from the variable sqlType. In the current form, sqlType=title is specified exclusively in the HTTP address, but perhaps in an extension of find.php another search would be possible.

Three possible search criteria can be passed in the form of URL variables: authID (title by a particular author), publID (title from a particular publisher), or search (title with particular initial letters).

In the search for titles by a particular author, a query first determines and then displays the name of the author. This query also serves for validation of authID. If the query returns no result (or more than one), then there is something other than a unique valid ID value to be dealt with. In this case, a brief error message is displayed. Otherwise, the name of the author is displayed in the "usual" way of writing it (given name followed by family name), and an SQL command is created according to the following format:

CREATE TEMPORARY TABLE tmpTitleIDs TYPE=HEAP
SELECT titles.titleID
FROM titles, rel_title_author
WHERE rel_title_author.authID = '$authID'
  AND titles.titleID = rel_title_author.titleID
ORDER BY title LIMIT 21

The search for titles of a particular publisher is accomplished with almost the same pattern, but now no JOIN is necessary. The search for titles with particular initial letters proceeds as in the similar case involving form data:

elseif($sqlType=="title") {// else to: if(  . . .    )  {sql form data }
    // authID contains a value: Search for titles by this author
    if($authID) {
        $result =
  mysql_query("SELECT authName FROM authors " .
          "WHERE authID='$authID' LIMIT 2");
        if(mysql_num_rows($result)!=1)
  echo "<p>Sorry, ID number for author seems to be invalid.</p>\n";
        else {
  echo "<p>Titles written by <i>",
    htmlentities(last_name_last(mysql_result($result, 0, 0))),
    "</i>:\n";
  $sql = $sqlcreate .
    "SELECT titles.titleID " .
    "FROM titles JOIN rel_title_author " .
    "WHERE rel_title_author.authID = '$authID' " .
    "  AND titles.titleID = rel_title_author.titleID " .
    "ORDER BY title " $sqllimit;
        }
        mysql_free_result($result);
    }
    // publID contains a value: search for titles of this publisher
    elseif($publID) {
        $result =
  mysql_query("SELECT publName FROM publishers " .
          "WHERE publID='$publID' LIMIT 2");
        if(mysql_num_rows($result)!=1)
  echo "<p>Sorry, ID number for publisher seems to be ",
        "invalid.</p>\n";
        else {
  echo "<p>Titles published by <i>",
    mysql_result($result, 0, 0), "</i>:</p>\n";
  $sql = $sqlcreate .
    "SELECT titleID FROM titles " .
    "WHERE publID = '$publID' " .
    "ORDER BY title "$sqllimit;
        }
        mysql_free_result($result);
    }
    //  search  contains search character string: search for titles with these
    // initial letters
    elseif($search) {
        echo "<p>Titles beginning with <i>",
  htmlentities($search), "</i>:</p>\n";
   $sql = $sqlcreate .
      "SELECT titleID FROM titles " .
      "WHERE title LIKE '" . addslashes($search) . "%' " .
      "ORDER BY title " .  $sqllimit;
    }
    if($page>1)
        echo "Page $page</p>\n";
} //  end of elseif($sqlType=="title")

Displaying Results of an Author Search

The last part of find.php is responsible for executing the command stored in sql and displaying the result. This is relatively uncomplicated when the search is for authors. The names are transformed with last_name_last into a more readable format. Furthermore, build_href is used to represent the name as an HTTP link. Here, sqlType= title and authID=n are passed as search strings.

if(!empty($sql)) {
  if($sqlType=="author") {
    // display results for author search
    $result = mysql_query($sql);
    if(!$result or !mysql_num_rows($result))
  echo "<p>No results.</p>\n";
        else {
  echo "<hr /><ul>\n";
  while($row = mysql_fetch_object($result)) {
    echo "<li>",
    build_href("./find.php",
      "sqlType=title&authID=$row->authID",
      last_name_last($row->authName)),
    "</li>\n";
  }
  echo "</ul>\n";
  }
}

Displaying Results of a Title Search

From the point of view of database programming, the code for the representation of book titles is much more interesting. With DROP TABLE, the table tmpTitleIDs is deleted (which can exist only if the script was previously executed and did not terminate due to an error). Then the table is rebuilt with the command contained in sql. If the query contained in sql has returned results, two additional queries are executed. The first determines the detailed information for each title:

SELECT titles.titleID AS titleID, titles.title AS title,
    titles.subtitle AS subtitle, titles.edition AS edition,
    titles.year, titles.isbn, titles.comment,
    publishers.publName AS publisher, publishers.publID,
    categories.catName AS category, languages.langName AS language
FROM titles JOIN tmpTitleIDs
LEFT JOIN categories ON titles.catID = categories.catID
LEFT JOIN languages ON titles.langID = languages.langID
LEFT JOIN publishers ON titles.publID = publishers.publID
WHERE titles.titleID = tmpTitleIDs.titleID

A result is the unification of data from five tables. In three cases, a LEFT JOIN is used, so that title records are created even if one of more of the fields catID, langID, and publID contain the value NULL.

elseif($sqlType=="title") {
    // there should not be any tmpTitleIDs---but
    mysql_query("DROP TABLE IF EXISTS tmpTitleIDs");
    // create temporary table tmpTitleIDs
    mysql_query($sql);
    if(!mysql_affected_rows()) {
        echo "<p>No results.</p>\n";
    }
    else {
        // query for the detailed title data
        $result1 = mysql_query
     ("SELECT titles.titleID AS titleID, " .
      . . .    (see above)
    "WHERE titles.titleID = tmpTitleIDs.titleID");

The second query determines a list of all titleID values and the associated author names. Please observe that this list can contain several entries for each titleID value (since there can be several authors for a given title):

SELECT tmpTitleIDs.titleID, rel_title_author.authID,
    authName AS author
FROM authors, rel_title_author, tmpTitleIDs
WHERE authors.authID = rel_title_author.authID
  AND rel_title_author.titleID = tmpTitleIDs.titleID
ORDER BY rel_title_author.authNr, authName

In the query, an n : m relation is created between the titleID in the tmpTitleIDs table and the authID values in the author table. The entries are sorted by authNr and authName. Thereby it is achieved that in the latter formation of the author list, the first author is indeed given first for each title. Only if no differentiation is possible based on authNr (because, for example, the field contains NULL) is an alphabetical order imposed:

// query for authors $result2 = mysql_query
   ("SELECT tmpTitleIDs.titleID, rel_title_author.authID, " .
   ... (see above)
   "ORDER BY rel_title_author.authNr");

The results contained in result2 are now brought into an array, where titleID is used as key. If there are several authors for a titleID, then the names are separated by commas. The author names are formatted with build_href as HTTP links, which later provide the user with a convenient way to search for all the titles of this author:

// enter author names in the field authors[]
while($row = mysql_fetch_object($result2)) {
      // separate authors by commas
      if($authors[$row->titleID])
         $authors[$row->titleID] .= ", ";
      $tmp = build_href("./find.php",
               "sqlType=title&authID=$row->authID",
               last_name_last($row->author));
      $authors[$row->titleID] .= $tmp;
}

After all data have been collected and the author list has been prepared, we may begin with the output of titles. In publref a publisher name is formatted as an HTTP link (similar to what was just done with author names). Then comes an almost interminable echo command, which looks quite complicated, because most of the title fields can be empty. Therefore, the PHP construction condition ? result1 : result2 is used several times. The expression returns result1 if condition is satisfied (is not 0); otherwise, result2.

// display title
echo "<hr /><ul>\n";
$titlecount=0;
while($row = mysql_fetch_object($result1)) {
      $titlecount++;
      if($titlecount<=$pagesize) {
    if($row->publisher)
      $publref = build_href("./find.php",
        "sqlType=title&publID=$row->publID",
        $row->publisher);
    echo "<p><li>", $authors[$row->titleID], ": ",
      "<b>", htmlentities($row->title), "</b>",
      $row->subtitle ?
        ", " . htmlentities($row->subtitle) . " " : "",
      $row->publisher || $row->year ? "<br />" : "",
      $row->publisher ? $publref : "",
      $row->year ? ", " .  htmlentities($row->year) . " " : "",
      $row->edition ?
        "<br />Edition: " .  htmlentities($row->edition) .
        " " : "",
      $row->isbn ?
        "<br />ISBN: " .  htmlentities($row->isbn) . " " : "",
      $row->language ?
        "<br />Language: <i>" . htmlentities($row->language) .
        "</i> " : "",
      $row->category ?
         "<br />Category: <i>" . htmlentities($row->category) .
         "</i> " : "",
      $row->comment ?
         "<br />Comment: <i>" . htmlentities($row->comment) .
         "</i> " : "",
      "</li></p>\n";
      }
}
echo "</ul>\n";

All that remains is to delete the temporary table tmpTitleIDs:

} //  end of the else block to if(!mysql_affected_rows())
// delete temporary table
mysql_query("DROP TABLE IF EXISTS tmpTitleIDs");

Links to Previous Pages and to the Next Page

If the displayed page number is greater than 1 or if more titles were found than were displayed ($titlecount>$pagesize), then a row with links to additional pages with search results is displayed (see Figure 12-4).

Click To expand
Figure 12-4: Links to additional pages with search results

Since for reasons of efficiency the total number of search results is not determined, near the top, only a link to the next page ($page+1)is displayed. Near the bottom, links to pages 1 through ($page-1) are displayed. Since the page $page is currently being shown, it does not make sense to display a link to this page. This number of pages is placed in parentheses for clarity. In addition to the page numbers, the two links PREVIOUS PAGE and NEXT PAGE are displayed.

Note that the PHP function urlencode is used in generating the page link for the part search=searchpattern, so that any blank or special characters contained in the search string will not cause problems:

// display links to additional search results
echo "<hr />\n";
if(isset($page) or $titlecount>$pagesize) {
  $query = "sqlType=title";
  $query .= isset($authID) ? "&authID=$authID" : "";
  $query .= isset($publID) ? "&publID=$publID" : "";
  $query .= isset($search) ?
    "&search=" . urlencode($search) : "";
  echo "<p>More results: ";
  // links to previous pages
  if(isset($page) and $page>1) {
    echo build_href("find.php",
          $query . "&page=" . ($page-1),
          "Previous page");
    echo " / Page ";
    for($i=1; $i<$page; $i++) {
      if($i>1) echo " ";
      echo " ",
        build_href("find.php", $query . "&page=" . $i, $i);
    }
  }
  // place current page in parentheses
  if($page>1) echo " ";
  else echo "Page ";
  echo "<b>($page)</b> ";
  // links to the next page
  if($titlecount>$pagesize) {
    if(isset($page)) echo " ";
    else $page=1;
    echo build_href("find.php",
          $query . "&page=" . ($page+1),
           ($page+1)), " / ";
    echo build_href("./find.php",
          $query . "&page=" . ($page+1),
          "Next page");
  }
  echo "</p>\n";
     }
  }
}

Ideas for Improvements

The search options in this example are rather limited. The following list contains some ideas for improvements:

  • Search criteria: Search for all books from publisher x that appeared between year1 and year2.Limit the search results to books in a particular language.

  • Category search: Search for all books in a given category and all subcategories.

  • Sort possibility: Sort by year of publication, category, publisher, etc. (In find.php sorting is always by title.)

  • Full-text search: Search by words that appear in the title, subtitle, commentary, or list of authors.


Simple Input of New Books (simpleinput.php)

This section shows a minimalist input form for new books (see Figure 12-5). A new book title together with a list of authors can be specified in only two input fields. Then the script tests whether any of the authors input are already stored in the database. All as yet unknown authors are inserted with INSERT. Then the new title is stored, as well as the required entries in rel_title_author for linking a title and its authors.

Click To expand
Figure 12-5: Simple book title input

The script simpleinput.php does not enable input of subtitles, publishers, category, and so on, and it is as good as useless in support of input error correction. The input of special characters can lead to problems. Boy, are we in trouble! Therefore, in the next section, where we introduce the script input.php, we will add all of these missing features as well as increase the convenience factor in input and allow for error correction.

The great advantage of simpleinput.php vis-à-vis input.php is its simplicity. The code is readable and easy to understand. Once you have understood simpleinput.php, you have a good foundation for what lies ahead, namely, wading through the much more extensive and sinuous highways and byways of input.php.

Program Structure, Code Outline

The structure of the script simpleinput.php is relatively simple. First some auxiliary functions are loaded from mylibraryconnect.inc.php, and the connection to the database is established. The next lines begin the resulting HTML document.

The script is responsible for both the display of the form and the evaluation of the form data. Therefore, the section Main-Code begins with the evaluation of several form variables. If these variables are empty, then all that happens is that the next form is displayed. If the input is incomplete, an error message is dispatched, and the form is displayed again so that input can be completed. If correct input is offered, then the data are stored in the database. In this case, too, an empty form is again displayed, so that additional data can be input.

<?php   // file mylibrary/simpleinput.php
    include("private/mylibraryconnect.inc.php");
    // read form variables (necessary only with PHP >= 4.1)
          $formSubmit = array_item($_POST, 'formSubmit');
    $formTitle     = array_item($_POST, 'formTitle');
    $formAuthors = array_item($_POST, 'formAuthors');
    // connection to database
    $connID = connect_to_mylibrary();
?>
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN">
<html><head>
    <meta http-equiv="Content-Type"
          content="text/html; charset=iso-8859-1" />
<title>PHP Programming: Input new titles (mylibrary database)</title>
</head><body>
<h2>Input a new title for the mylibrary database</h2>
<?php
    // Main-Code
    //evaluation of the form, store new record if any
        . . .
?>
<!--  Form  -->
<form method="post" action="simpleinput.php">
<p>Title:
    <br />
    <input name="formTitle" size=60 maxlength=80
    value="<?php echo htmlspecialchars($formTitle); ?>" /></p>
    <p>Authors:
    <br /><input name="formAuthors" size=60 maxlength=100
            value="<?php echo htmlspecialchars($formAuthors); ?>" />
    <br />(Last name first! If you want to specify more than one
    author, use ; to seperate them!)</p>
    <p><input type="submit" value="OK" name="formSubmit" /></p>
</form>
</body></html>

Form Variables

When the script for the evaluation of the form input is invoked, three variables receive the contents of the form fields:

  • formSubmit contains "OK".

  • formTitle contains the title of the book to be stored.

  • formAuthors contains the semicolon-separated list of authors.

Store a Title

The code for evaluating the form is executed only if the form variables contain data. In this case, it is first checked whether the input is complete, that is, whether both a title and author(s) were input. If that is not the case, then a brief error message is displayed and the form is again displayed with the partial data. The variables formTitle and formAuthors are then freed of any magic quote characters, so that the form is ready for additional input:

if($formSubmit) {
    // check for completeness of the form
    if(empty($formTitle) or empty($formAuthors)) {
        echo "<p>Please specify title and at least one author.</p>\n";
        // remove magic quotes
        if(get_magic_quotes_gpc()) {
    $formTitle = stripslashes($formTitle);
    $formAuthors = stripslashes($formAuthors);
        } }

The rest of the code, for storing the new record, is contained in the ensuing else block. The author character string is transformed by explode into an array, which then is run through a loop. For each author name a check is made as to whether that author already appears in the database. If that is the case, then the ID number is stored in the array authIDs. Otherwise, the new author is inserted into the database with INSERT. The ID number of the author is then determined with the function mysql_insert_id:

else {
    $authCount=0;
    $authorsArray = explode(";", $formAuthors);
    // loop over all authors
    while($i = each($authorsArray)) {
        $author = trim($i[1]);
        if($author) {
    // is the author already known to the database?
    $result =
            mysql_query("SELECT authID FROM authors " .
            "WHERE authName = '$author'");
    if(mysql_num_rows($result))
            $authIDs[$authCount++] = mysql_result($result, 0, 0);
    // author is not known --> insert author
    else {
            mysql_query("INSERT INTO authors (authName) " .
            "VALUES('$author')");
            $authIDs[$authCount++] = mysql_insert_id();
    } } } // end of the  author loop

A further INSERT command is then invoked to store the title. The resulting ID number is then used in constructing the SQL command by which the relation between the book title and its authors is established.

// store title
mysql_query("INSERT INTO titles (title) VALUES ('" .
      trim($formTitle) . "')");
$titleID=mysql_insert_id();
// establish title/author relation
$sql = "INSERT INTO rel_title_author " .
    "(titleID, authID, authNr) VALUES ";
for($i=0; $i<$authCount; $i++) {
    if($i!=0) $sql .= ",";
    $sql .= "($titleID, $authIDs[$i], $i)";
}
mysql_query($sql);

The SQL command has the following form:

INSERT INTO rel_title_author (titleID, authID, authNr)
VALUES (titleID, authorID1, 1), (titleID, authorID2, 2) ...

To finish, a message is displayed (in a very sloppy manner, without checking the return value mysql_query) that the data were stored and that input of the next data record may now begin. The input just carried out can be checked thanks to a link to find.php. (There, all titles whose names begin like that of the just-stored book are displayed.) In order that an empty form be displayed for the next input, the variables formTitle and formAuthors are cleared:

  echo "<p>Your last input has been saved and can be seen ",
  build_href("find.php",
    "sqlType=title&search=" . trim($formTitle),
    "here"),
  ".\n",
<br />You may now continue with the next title.</p>\n";
    $formTitle = $formAuthors = "";
  } //  end of the else block for storing data
} //  end of the if block for form evaluation


Convenient Input of New Book Data (input.php)

The input form introduced in the last section was a minimalist enterprise in every sense. (If it were music, it would be by Philip Glass.) The script input.php offers more in the way of input possibilities, more convenience, and more data security. The price for all of these added features is rather more complex code, that is, code that is closer to a real-world application than was our previous example simpleinput.php.

Operating Instructions

Title input takes place in two phases. In the first phase, only the initial letters of the author and publisher names may be input (see Figure 12-6).

Click To expand
Figure 12-6: Input of a new book title (phase 1)

The script input.php searches the database for similar names and then in the second phase presents them as suggestions (see Figure 12-7). This can save the user a bit of typing, but the true benefit is the possible prevention of adding misspelled names to the database.

The script input.php expects author names in the customary (unless you are Hungarian, Chinese, ... ) order of given name followed by family name, and then it takes care of storing it in the form specific to the database, namely, family name, then given name.

The script recognizes existing authors even if the name is input in the incorrect order. Thus, for example, if you input (horror of horrors!) King Stephen instead of Stephen King and this particular author happens to be in the database already, then the program recognizes the existing name and does not store the "new" name, this time in the incorrect order. Needless to say, this technique of error correction does not work with new names. If the given and family names of an author are incorrectly input in reverse order, that author will always be displayed incorrectly in the future. (Nonetheless, the name is false in a consistent way, and can easily be corrected. There are no duplicates in the sense that the name is stored twice, once as "Stephen King", and once as "King Stephen".)

From the data input in Figure 12-6, the script has correctly recognized the first author (Simon Garfinkel) and the publisher from the initial letters. The second author (Gene Spafford) was not found in the database, and so this name must be input in full.

Click To expand
Figure 12-7: Checking the input (phase 2)

In addition, the first validation of the input now takes place. Was an integer greater than zero given as edition number? Is the year of publication believable (that is, not greater than the current year plus 1)? If input errors are suspected, the HTML document displays red warnings, such as, for example "Publishing year needs to be either empty or a 4-digit integer number not larger than 2002."

The validation process is repeated in connection with phase 2. Furthermore, it is checked whether an option was selected in the case that the user had a choice in the selection of the publisher or author names. If errors are detected, then an error message is displayed and the form is displayed again so that corrections can be entered.

Program Structure

The script input.php includes about one thousand lines of code. The first two-thirds of this code contains definitions of numerous functions that are called in the last third (main code). The code of most of the functions is described in this section. But first let us have a bit of an overview to orient ourselves:

  • sql_str prepares a character string for use in an SQL command (addslashes, within single quotation marks).

  • str_or_null returns NULL or an SQL-suitable character string (sql_str($x)).

  • num_or_null returns NULL or an integer.

  • last_name_first("John F. Kennedy") returns "Kennedy John F.". The function transforms a name from given name first format to the database-specific format of given name last.

  • last_name_last("Kennedy John F.") returns "John F. Kennedy". This represents the inverse function to last_name_first.

  • build_select_list generates a selection list (<select>) for an HTML form from an SQL query.

  • build_categories_select generates a hierarchical selection list for an HTML form from the categories table.

  • show_publisher_options evaluates the publisher name and displays a list of similar names in the form. The function is called between input phases 1 and 2.

  • restore_publisher_options again displays the publisher list. The function called in input phase 2 must be repeated due to input errors.

  • show_authors_options and restore_authors_options fulfill the analogous tasks for the author names.

  • validate_stage1 and validate_stage2 check the input data and display error messages as required.

  • save_data stores the input data in the database.

Code Outline

The main code begins with an evaluation of the form data (if such are available). Depending on the result, these data are again displayed (for completion or correction), or the data are stored and a new form is displayed for input of the next data record:

<!-- php/mylibrary/input.php -->
<?php
// read in auxiliary functions
include("../../_private/mylibraryconnect.inc.php");
// define new functions
function sql_str($x) { . . . }
function str_or_null($x) { . . . }
function num_or_null($x) { . . . }
function last_name_first($x) { . . . }
function last_name_last($x) { . . . }
function build_select_list($formname, $sql, $defaultitem) { . . . }
function build_categories_select($defaultitem) { . . . }
function show_publisher_options($publname) { . . . }
function restore_publisher_options($formvars) { . . . }
function show_authors_options($formAuthors) { . . . }
function restore_authors_options($formvars) { . . . }
function validate_stage1($formvars) { . . . }
function validate_stage2($formvars) { . . . }
function save_data($formvars) { . . . }
// establish connection to the database
$connID = connect_to_mylibrary();
?>
<!-- HTML-Header -->
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN">
<html><head>
<title>PHP Programming, input title (mylibrary database)</title>
</head><body>
<h2>Input new title for the mylibrary database</h2>
<?php
// outline of the program structure
//eliminate \', \", \\ and \0 from form variables
// $stage=1;
// if there are form data from phase 1:
// validate data;
// $stage=2; (--> Phase 2)
// if there are form data from phase 2:
// validate data;
// if OK: store data; $stage=1; (--> next input)
// otherwise: display error message, $stage=2; (--> correction)
// if $stage==1:
// display form phase 1
// otherwise
// display form phase 2
?>

Form Variables: Phase 1

If the script is called with form data from phase 1, then the following variables are assigned values:

  • formSubmit1 contains "OK".

  • formTitle and formSubtitle contain book title and subtitle.

  • formAuthors contains a comma- or semicolon-separated list of author names.

  • formEdition contains the edition number.

  • formCategory contains the ID number of the selected category.

  • formYear contains the publication year of the book.

  • formLanguage contains the ID number of the language of the book.

  • formISBN contains the ISBN number.

  • formComment contains a comment or key words.

All variables except for formAuthors and formTitle may be empty.

Form Variables: Phase 2

  • formSubmit2 contains "OK".

  • formTitle, formSubtitle, formEdition, formCategory, formYear, formLanguage, formISBN, formComment have the same meaning as above.

  • formAuthorsCount specifies the number of authors displayed.

  • formAuthorType<n> specifies how author n was handled: 1 means that the author is already known; 2 means that the author is unknown and no similar author names were found; 3 means that the author was not found exactly, but one or more authors with the same initial letters were found. These authors are displayed as options.

  • formAuthor<n> contains the input name of n.

  • formAuthorRadio<n> contains either "new", if the radio button by the text field was clicked, or the ID number of an existing author if the option for that author was clicked (only for the case formAuthorType<n>=3).

  • formAuthorNames<n> contains a semicolon-separated list of all authors found (only in the case formAuthorType<n>=3).

  • formAuthorIDs<n> contains a semicolon-separated list of all found authors (only in the case formAuthorType<n>=3).

  • formPublType tells whether a publisher was specified and whether this publisher is already known: 0 means no publisher; 1 means known publisher; 2 means unknown publisher; 3 means a list of publishers.

  • formPublisher contains the name of the publisher.

  • formPublRadio contains either "new" or the ID number of an existing publisher (only in the case formAuthorType<n>=3).

  • formPublNames and formPublIDs have the same meaning as formAuthorNames and IDs.

Using Form Variables

The backslashes before single and double quotation marks that are usually inserted by PHP have proved to be troublesome in this program. For the time-consuming evaluation of a form and generation of new forms, it is burdensome always to be thinking about stripslashes. Therefore, the following loop at the start of code execution (main code)removes magic quotes from all form variables:

if(get_magic_quotes_gpc()) {
  while($i = each($_POST)) {
    $_POST[$i[0]] = stripslashes($i[1]);
  }
}

Then all form data are read form the $_POST field and stored in variables. The function item_array, described already several times in this book, is defined in mylibraryconnect.inc.php:

$formSubmit1         = array_item($_POST, 'formSubmit1');
$formTitle           = array_item($_POST, 'formTitle');
// etc. fŸr $formSubtitle, $formAuthors, $formEdition, $formCategory,
// $formYear, $formLanguage, $formISBN, $formComment, $formSubmit2,
// $formAuthorsCount, $formPublisher, $formPublType, $formPublRadio
// $formPublNames and $formPublIDs

The form data for authors requires special treatment, since their number is variable. You could actually encounter a book in the form that was written by ten authors. In such a case there are the variables formAuthor1, formAuthor2, through formAuthor10. Reading into these variables is done in a loop in which the variable names are created dynamically using an instruction of the form ${'name' . $n}, where $n contains a number:

for($authCount=1; $authCount<=$formAuthorsCount; $authCount++) {
  ${'formAuthorType' . $authCount} =
    array_item($_POST, 'formAuthorType' . $authCount);
  ${'formAuthor' . $authCount} =
    array_item($_POST, 'formAuthor' . $authCount);
  ${'formAuthorRadio' . $authCount} =
    array_item($_POST, 'formAuthorRadio' . $authCount);
  ${'formAuthorNames' . $authCount} =
    array_item($_POST, 'formAuthorNames' . $authCount);
  ${'formAuthorIDs' . $authCount} =
    array_item($_POST, 'formAuthorIDs' . $authCount);
}

Special Features of PHP

The two PHP functions htmlentities and htmlspecialchars help in representing special characters correctly. htmlspecialchars changes only the characters ", &, <, >, while htmlentities also changes a number of other characters, such as ü,é, à.

  • In input.php, htmlspecialchars is always used if existing data is to be reinserted into a text field of a form. (In this case, htmlentities is unsuitable, since then constructions like &auml; appear in character strings that are to be stored later in the database).

  • htmlentities,on the other hand, is used when data are to be displayed unaltered (thus not within text input fields in a form). In fact, htmlentities should be superfluous, since in any case, ISO-8559-1 is used as the character set for the HTML file (that is, the latin1 character set, which handles a number of European special characters). htmlentities, however, increases the probability that special characters will be displayed correctly, independent of the version and configuration of the web browser and of the PHP configuration (php.ini).

With the PHP command echo, character strings can be placed within single or double quotation marks. Each of these variants has its advantages, and both are frequently used. If $variable has the value 123, then the following hold:

echo "xy='$variable' \n" outputs the character string xy='123' followed by a line break character. Thus echo " ... " supports the automatic placing of variable contents, the output of the character ', and the output of special characters such as \n and \t. The double quote character can be represented by \",asin echo "x=\"abc\"".

  • echo 'xy="$variable"\n' outputs the string xy="$variable"\n; thus echo ' ... ' enables the direct output of the character " needed in HTML code. In general, everything is output unchanged in single quotation marks (no variable substitution, no support of special characters like \n).

  • In the code of input.php, the ' ... ' and ... parts are often run together, for example thus:

    echo '<input type="hidden" ', "name=$typename ", 'value="2" />', "\n";

Data Validation, Changeover for Input Phases 1 and 2

Depending on whether form data are present from phase 1 or phase 2, a more or less thorough validation of the data and their eventual storage takes place. The result of the validation process determines the content of the variable stage. This variable then determines in the rest of the program how the formula is displayed.

If data from phase 1 are present, then it is ensured that at least the author field has been filled in. If that is the case, then a changeover to phase 2 takes place. Otherwise, the input form for phase 1 is displayed anew:

// if $formSubmit1 and $formSubmit2 are empty
$stage=1;
// evaluation of form data from phase 1
if($formSubmit1) {
  // in the author list replace, by ;
  $formAuthors = str_replace(",", ";", $formAuthors);
  // check whether formAuthor contains data (other than ";" and " "):
  // if formAuthor is empty, remain at phase 2
  if(!trim(str_replace(";", "", $formAuthors))) {
    echo '<p><font color="#ff0000">You must specify ',
  "<i>authors</i> in stage 1.</font></p>\n";
    $stage=1;
  }
  // otherwise, change over to phase 2
  // possible display of error messages via validate_stage1()
  else {
    validate_stage1();
    $stage=2;
  }
}

If the form data come from phase 2, then the function validate_stage2 is summoned. If no errors occur, then the data are stored with save_data. Then all of the input variables are reset. First stage is reset to 1; that is, an empty form is displayed for input of the next book title. If errors occurred in the validation process, then it remains at stage=2; that is, the form is displayed again in its current state.

// evaluation of form data from phase 2
if($formSubmit2) {
  // basic validation; then store data
  // or display input form again for corrections
  if(validate_stage2($HTTP_POST_VARS)) {
    save_data($HTTP_POST_VARS);
    echo "<p>Last input has been saved.\n";
    // input of next record
    $stage=1;
    // reset form variables
    $formTitle = $formSubtitle = $formYear = $formISBN="";
    $formAuthors = $formPublisher = $formCategory = "";
    $formComment = $formLanguage = $formEdition = "";
  }  else
    // display form again for corrections
    $stage=2;
}

Validating Data (validate_stage1 and validate_stage2)

In validate_stage1 it is ensured that a title is specified, that the input field for the query either is empty or contains a number greater than 0, and that the input field for the publication year either is empty or contains a number that is not greater than one plus the current year. (It is typical for a book that appears in the last three months of the year to bear the next year as publication date.)

The function validate_stage2 relies on validate_stage1. Moreover, the function checks the input for author and publisher names. It must be ensured that either a string was specified or an option selected. To save space, here we present only the code of validate_stage1:

Remark

No parameters are passed to the functions validate_stage1 and validate_stage2. Instead, at the beginning of these functions the necessary variables are declared as global, so that they can be accessed. In validate_stage2, this is a bit more complicated in that depending on the number of authors, there can be a variable number of formAuthorXxx variables:

for($authCount=1; $authCount<=$formAuthorsCount; $authCount++) global
${'formAuthorType' . $authCount},
${'formAuthor' . $authCount},
${'formAuthorRadio' . $authCount}, ${'formAuthorNames' . $authCount},
${'formAuthorIDs' . $authCount};
function  validate_stage1() {
global $formTitle, $formEdition, $formYear;
  $errstart = '<br /><font color="#ff0000">';
  $errend = "</font>\n";
  $errors = 0;
  echo "<p>";
  // check title
  if(!trim($formTitle)) {
    echo $errstart, "Please enter an <i>title</i>.\n", $errend;
    $errors++;
  }
  // check publisher
  if($formEdition and
   (!is_numeric($formEdition) or intval($formEdition)<1)) {
    echo $errstart, "<i>Edition</i> needs to be either empty or an   ",
    "integer greater than or equal to 1.", $errend;
    $errors++;
  }
  // check year of publication; may not be greater
  // than the current year + 1
  // (31536000 is the number of seconds in a year)
  if($formYear and
   (!is_numeric($formYear) or
    intval($formYear) > date("Y", time()+31536000))) {
    echo $errstart, "<i>Publishing year</i> needs to be either ",
    "empty or a 4-digit integer number not larger than ",
    date("Y", time()+31536000), ".", $errend;
    $errors++;
  }
  echo "</p>\n";
    return($errors);
}

Displaying Forms

Displaying the Form for Input Phase 1

The HTML code for the form is given directly, for the most part. There are only a few embedded PHP functions for displaying the content of variables and creating the hierarchical category listbox (function build_categories_select) and the language listbox (function build_select_list). The code of both these functions will appear later.

Within the form, a table is used to achieve a somewhat pleasing appearance. In the following, only the more interesting code lines are given, where the PHP code embedded in the HTML code is printed in boldface:

if($stage==1) {
?>
<h3>Input new title (stage 1)</h3>
<form method="POST" action="input.php">
<table>
<tr><td>Title: *</td>
    <td><input name="formTitle" size="60" maxlength="80"
      value="<?php echo htmlspecialchars($formTitle); ?>" />
</td></tr>
<tr><td>Subtitle:  . .    (as above)
<tr><td>Edition:  . .
<tr><td>Authors: *</td>
    <td><input name="formAuthors" size="60" maxlength="100"
      value="<?php echo htmlspecialchars($formAuthors); ?>" />
</td></tr>
<tr><td>Publisher:</td>
    <td><input name="formPublisher" size="60" maxlength="100"
      value="<?php echo htmlspecialchars($formPublisher); ?>" />
</td></tr>
<tr><td>Category:</td>
      <td><?php build_categories_select($formCategory);  ?></td></tr>
<tr><td>Publishing year:  . .
<tr><td>Language:</td>
    <td><?php build_select_list("formLanguage",
      "SELECT langID, langName FROM languages " .
      "ORDER BY langName", $formLanguage); ?></td></tr>
<tr><td>ISBN:  . .
<tr><td>Comment/Keywords:  . .
<tr><td>
      <td><input type="submit" value="OK" name="formSubmit1"></tr>
</table>
</form>

Displaying the Form for Input Phase 2

For most of the input fields the code for input phase 2 is simply a repetition of that for phase 1. What is new is the processing of authors and publisher. If a form from phase 1 was just displayed (that is, if($formSubmit1)), then the input is evaluated with show_authors_options or show_publisher_options. Similar names in the database are then displayed in the form in radio button fields.

If just previously, a form in phase 2 was displayed (during the evaluation of which input errors were detected that now must be corrected), then the radio button fields from the form just displayed must be restored. The functions restore_authors_options and restore_publishers_options access the data that have been stored in hidden fields of the form:

<?php
if($stage==2) {
?>
<h3>Verify data (stage 2)</h3>
<form method="post" action="input.php">
<table cellpadding="5">
<tr><td>Title: *</td>
  <td><input name="formTitle" size="60" maxlength="80"
             value="<?php echo htmlspecialchars($formTitle); ?>" />
</td></tr>
<tr><td>Subtitle: . . .
<tr><td>Edition: . . .
<?php
if($formSubmit1)
  show_authors_options($formAuthors);
else
  restore_authors_options();
if($formSubmit1)
  show_publisher_options($formPublisher);
else
  restore_publisher_options();
?>
<tr><td>Category: . . .
<tr><td>Publishing year: . . .
<tr><td>Language: . . .
<tr><td>ISBN: . . .
<tr><td>Comment/Keywords: . . .
<tr><td>
   <td><input type="submit" value="OK" name="formSubmit2"></tr>
</table>
</form>

Auxiliary Functions

SQL Auxiliary Functions (sql_str, str_or_null, num_or_null)

In formulating SQL commands, character strings must always be given in single quotation marks. Special characters must be preceded by a backslash. Distinction must be made between a number or character string and NULL according to whether or not a PHP variable is empty. The following three functions are helpful in this regard:

//mark ', ", and \, place character strings in '
function  sql_str($x) {
    return "'" . addslashes($x) . "'";
}
// returns SQL-suitable character string or NULL
function  str_or_null($x) {
  if(trim($x)=="")
    return("NULL");
  else
      return(sql_str(trim($x)));
}
// returns integer or NULL
function  num_or_null($x) {
  if(empty($x))
    return("NULL");
  else
    return($x);
}

Functions for Name Processing (last_name_first, last_name_last)

In databases, names must generally be stored with family name first so that searching and sorting by family name can proceed in as simple a manner as possible. However, people are not, at least the last time we checked, computers. In many cultures they are accustomed to writing given name first, and they are inclined to write the name in that order.

The function last_name_first assumes that the character string passed as parameter contains a name in the given-name-first format. This function places the family name (the part of the character string after the last space character) at the beginning of the string. The string is unchanged if it consists of a single word or if it ends with a period. (Since usually, only given names are abbreviated with a period, in this case the assumption is made that the character string already appears in the form suitable for storage in the database. This safety measure is designed to recognize names that are input in the form in the incorrect order.)

// "John F. Kennedy" --> "Kennedy John F."
function  last_name_first($x) {
  $x = trim($x);
  // no change if ends with a period
  if(substr($x, -1) == ".") return($x);
  // no change if there is no space character
  if(!$pos = strrpos($x, " ")) return($x);
  // place last word at the head of the character string
  return(trim(substr($x, $pos+1) . " " .  substr($x, 0, $pos)));
}

The function last_name_last is more or less the inverse function. As parameter is passed a character string in the database-specific ordering (with family name at the head). The first word is then placed at the end of the string.

function  last_name_last($x) {
  $x = trim($x);
  // no change if there is no space character
  if(!$pos = strpos($x, " ")) return($x);
  // place first word at the end of the string
  return(trim(substr($x, $pos+1) . " " .  substr($x, 0, $pos)));
}

build_select_list

The function build_select_list expects as its first parameter the name of the listbox, and as the second parameter it wants an SQL query, while for the third parameter the ID number of the default entry for the listbox is required. The result of the SQL query must contain the ID value in the first column, and in the second, a character string from the table (here, for example, the name of the language). From this information the function creates the HTML code for the listbox (see Figure 12-8). The following lines show how the function is called:

build_select_list("formLanguage",
  "SELECT langID, langName FROM languages " .
  "ORDER BY langName", "");

The resulting HTML code looks as follows. Please note especially that although the character strings are displayed (in Figure 12-8 the names of languages), it is their ID numbers that are stored with value= ... . The ID numbers are much better suited than character strings for further processing of the selection:

Figure 12-8: Listbox for languages
<select name="formLanguage" size="1">
<option value="none">(choose)</option>
<option value="2"> deutsch</option>
<option value="1"> english</option>
<option value="4"> norsk</option>
<option value="3"> svensk</option>
</select>
function  build_categories_select($defaultitem) {
  $result =
    mysql_query("SELECT catID, catName, hierIndent " .
    "FROM categories " .
    "ORDER BY hierNr");
  echo '<select name="formCategory" size="1">', "\n";
  echo '<option value="none">(choose)</option>', "\n";
  while($row=mysql_fetch_object($result)) {
    echo "<option ";
    if($defaultitem==$row->catID) echo 'selected="selected" ';
    echo "value=\"$row->catID\">",
  $row->hierIndent>0 ?
    str_repeat("&nbsp;&nbsp;&nbsp;", $row->hierIndent) : "",
  htmlentities($row->catName), "</option>\n";
  }
  echo "</select>\n";
  mysql_free_result($result);
}

Creating a Hierarchical Listbox (build_categories_select)

The function build_categories_select is related to the function just introduced, though it is less broadly applicable. The function evaluates the categories table and then creates a hierarchical listbox (that is, the list entries are indented according to their place in the hierarchy; see Figure 12-9). For indentation, nonbreaking space characters are used (HTML code &nbsp;). As a single parameter, the ID number of a default category can be passed to the function:

Figure 12-9: Hierarchical listbox for selecting the book category
function  build_categories_select($defaultitem) {
  $result =
    mysql_query("SELECT catID, catName, hierIndent " .
    "FROM categories " .
    "ORDER BY hierNr");
  echo '<select name="formCategory" size="1">', "\n";
  echo '<option value="none">(choose)</option>', "\n";
  while($row=mysql_fetch_object($result)) {
    echo "<option ";
    if($defaultitem==$row->catID) echo 'selected="selected" ';
    echo "value=\"$row->catID\">",
  $row->hierIndent>0 ?
    str_repeat("&nbsp;&nbsp;&nbsp;", $row->hierIndent) : "",
  htmlentities($row->catName), "</option>\n";
  }
  echo "</select>\n";
  mysql_free_result($result);
}

Publisher Name (show_publisher_options)

The function show_publisher_options is invoked to display the input fields for the publisher name in the second form. The parameter publname contains the publisher name provided in the first form. Now an attempt is made to find the name that was input in the publishers table. If this does not succeed, then a list of similar names is sought. For this purpose the character string is reduced to at most six characters. Then a SELECT query is used to find all publisher names that begin with these letters:

function  show_publisher_options($publname) {
  // if no name was given thus far, display an empty input field
  $publname=trim($publname);
  if($publname=="") {
    echo '<tr><td>Publisher:</td>', "\n",
  '<td><input name="formPublisher" size="60" maxlength="100" ',
  'value="" /></td></tr>', "\n",
  '<input type="hidden" name="formPublType" value="0" />', "\n";
    return;
  }
  // is this publisher known?
  $result = mysql_query("SELECT publName FROM publishers " .
        "WHERE publName = " .  sql_str($publname) . "  LIMIT 2");
  if(mysql_num_rows($result)==1) {
    $known_publisher = 1;
    $publname = mysql_result($result, 0, 0);
    mysql_free_result($result);
  }
  // search for similar names
  else {
    if(strlen($publname<7))
  $publpattern = $publname . '%';
    else
  $publpattern = substr($publname, 0, 6) . '%';
    $result =
  mysql_query("SELECT publID, publName FROM publishers " .
    "WHERE publName LIKE " .  sql_str($publpattern) .
    "  ORDER BY publName LIMIT 20");
  }

The result is displayed for this analytical phase. There are three variants:

  • The publisher was found. The name is displayed in an input field titled Known publisher.

  • The publisher was not found, and there were no similar names found as well. Apparently, we are dealing with a new publisher. The name is again shown in an input field, but this time the title is New publisher. Furthermore, a message is displayed to the effect that input must be in the correct format.

  • The publisher was not found, but there were found publishers with the same initial letters. In this case both an input field and radio button list with all publishers found are displayed. (The length of this list is limited by LIMIT 20 in the SELECT query.) The user now has the opportunity to specify a new publisher name in detail or to select one of the options offered (see Figure 12-10). To keep the form readable, in this case the options are displayed as a table:

    Click To expand
    Figure 12-10: Selection or new input of a publisher name in the second input phase

// display input field/radio-button options in form
// case 1: publisher is known
if(isset($known_publisher)) {
  echo "<tr><td>Known Publisher:</td>\n",
    '<td><input name="formPublisher" size="60" maxlength="100" ',
    'value="', htmlspecialchars($publname), '" />',
    "\n</td></tr>\n",
    '<input type="hidden" name="formPublType" value="1" />', "\n";
}
else {
  // case 2: publisher unknown, no similar publishers found
  if(!mysql_num_rows($result)) {
      echo "<tr><td>New Publisher:</td>\n",
  "<td>Please specify full name. Please use ",
  "correct spelling.\n",
  '<br /><input name="formPublisher" size="60" ',
  '  maxlength="100" value="', htmlspecialchars($publname),
  '" />', "\n</td></tr>\n",
  '<input type="hidden" name="formPublType" value="2" />', "\n";
  }
  // case 3: publisher unknown, but similar publishers found
  else {
      echo "<tr><td>Publisher:</td>\n",
  "<td><table cellpadding="2">\n";
    echo '<tr><td><input type="radio" name="formPublRadio" ',
  'value="new" /></td>',
  '<td>New Publisher (Please specify full name. Please use ',
  'correct spelling.)', "\n",
  '<input name="formPublisher" size="60" maxlength="100" ',
  'value="', htmlspecialchars($publname),
  '" /></td></tr>', "\n";
    // display list of similar publishers
    $publNames="";
    $publIDs="";
    while($row=mysql_fetch_object($result)) {
  echo '<tr><td><input type="radio" name="formPublRadio" ',
    mysql_num_rows($result)==1 ? 'checked="checked" ' : '',
    'value="', $row->publID, '" /></td>',
    '<td>', $row->publName, "</td></tr>\n";
    $publNames .= ";" . $row->publName;
    $publIDs .= ";" . $row->publID;
    }
    echo "</table></td>\n</tr>\n";
    // place information about input fields in hidden form fields
    echo '<input type="hidden" name="formPublType" value="3" />',
  "\n";
      echo '<input type="hidden" name="formPublNames" value="',
  htmlspecialchars(substr($publNames, 1)), '" />', "\n";
      echo '<input type="hidden" name="formPublIDs" value="',
  substr($publIDs, 1), '" />', "\n";
  }
  mysql_free_result($result);   }   }

At the end of the input, options information is stored in three hidden form fields as to whether the input name is known (formPublType). If options with similar publisher names were displayed, then a list of these names and ID numbers is stored. In this way the input form can be re-created if necessary with restore_publisher_options.(This will be necessary if the phase 2 form must be displayed again due to an input error having been discovered in one or another field.) The following lines show the HTML code corresponding to Figure 12-10, including the hidden fields:

<tr><td>Publisher:</td>
  <td>
  <table cellpadding="2">
  <tr>
  <td><input type="radio" name="formPublRadio" value="new" /></td>
    <td>New Publisher (Please specify full name. Please use
        correct spelling.)
    <input name="formPublisher" size=60 maxlength=100 value="a" />
  </td>
  </tr>
  <tr><td><input type="radio" name="formPublRadio" value="1" /></td>
    <td>Addison-Wesley</td></tr>
  <tr><td><input type=radio name="formPublRadio" value="2" /></td>
    <td>apress</td></tr>
  </table>
</td></tr>
<input type="hidden" name="formPublType" value="3" />
<input type="hidden" name="formPublNames"
       value="Addison-Wesley;apress" />
<input type=hidden name="formPublIDs" value="1;2" />

Restoring the Publisher Selection Fields (restore_publisher_options)

The function restore_publisher_options is called if the form displayed in phase 2 of the input process must be shown again (due to the discovery of an input error). The form variables formPublisher (from the input field) and formPublRadio (selected options), as well as formPublType, formPublNames, and formPublIDs (hidden fields), are used to restore the form to its last displayed condition.

From the point of view of database programming, the code is not particularly interesting, for which reason we shall refrain from presenting it.

Author Input (show_authors_options and restore_authors_options)

The functions show_authors_options and restore_authors_options take on the same tasks as the publisher functions that we have just described, although this time it is for the input of author names. There are two significant differences:

In the input form in phase 1, several authors can be input. Therefore, show_authors_options decomposes the character string formAuthors into its components and then treats each author separately.

  • With all the author names, the order of names is reversed. (The program expects input in the form Stephen King, and then it searches the database for King Stephen.) Just to be on the safe side, the author table searches for names in both forms, so that the author will be found even if the input was in the incorrect order:

    nction  show_authors_options($formAuthors) {
  // decompose formAuthors into individual author names
  $authArray = explode(";", $formAuthors);
  $authCount = 0;
  // loop over all the authors
  while($i = each($authArray)) {
    $known_author = 0;
    $author = trim($i[1]);
    if($author) {
  $authCount++;
  // author perhaps has the given name first
  // but authorDB is in database format
  $authorDB = last_name_first($author);
  // search for names (with authorDB)
  $result1 =
    mysql_query("SELECT authName FROM authors " .
    "WHERE authName = " .  sql_str($authorDB) .
    "LIMIT 2");
  if(mysql_num_rows($result1)==1) {
    $known_author=1;
    $authorDB = mysql_result($result1, 0, 0);
    $author = last_name_last($authorDB);
  }
  // search again for names (with author)
  else {
    $result2 =
    mysql_query("SELECT authName FROM authors " .
    "WHERE authName = " .  sql_str($author) .
    "LIMIT 2");
    . . .

The rest of the code for show_authors_options is very similar to that of show_publisher_options, for which reason we do not show it here. The same holds for restore_authors_options.

Storing Data

From the database programming viewpoint we are again in interesting territory with the function save_data. The data input in the form should now be stored in the database.

In the first step it is checked whether the input publisher already appears in the database. If this is the case, then the ID number of the record is temporarily stored in publID. (The ID number will be needed later to store the title data record.) If the publisher is as yet unknown, then an INSERT command is employed to insert it into the table. Now publID is determined via mysql_insert_id. Things are simpler if the user has clicked on one of the radio buttons. Then the form variable formPublRadio already contains the ID number. Finally, there is the possibility that no publisher was specified in the form. In that case, publID is deleted with unset. (The titles table allows the storage of records without a publisher. Likewise, most of the other columns of the titles table are optional, and may contain NULL.)

function  save_data() {
  global $formTitle, $formSubtitle, $formEdition,
  $formCategory, $formYear, $formLanguage, $formISBN,
  $formComment, $formAuthorsCount, $formPublisher, $formPublType,
  $formPublRadio, $formPublNames, $formPublIDs;
  for($authCount=1; $authCount<=$formAuthorsCount; $authCount++)
    global ${'formAuthorType' . $authCount},
  ${'formAuthor' . $authCount},
  ${'formAuthorRadio' . $authCount},
  ${'formAuthorNames' . $authCount},
  ${'formAuthorIDs' . $authCount};
  // search for or store publisher
  $formPublisher = trim($formPublisher);
  if(($formPublType<3 or
  ($formPublType==3 and $formPublRadio=="new"))
    and !empty($formPublisher)) {
    // is the publisher already known to the database?
    $result =
  mysql_query("SELECT publID FROM publishers " .
    "WHERE publName = " .  sql_str($formPublisher));
    if(mysql_num_rows($result))
  $publID = mysql_result($result, 0, 0);
    // publisher is not yet known; store it
    else {
  mysql_query("INSERT INTO publishers (publName) " .
    "VALUES(" . sql_str($formPublisher) . ")");
  $publID = mysql_insert_id();
    }
    mysql_free_result($result);
  }
  // radio button with known publisher was selected
  elseif($formPublType==3 and $formPublRadio!="new") {
      $publID = $formPublRadio;
  }
  // no publisher was specified
  else
  ($publID = "";

A similar process is repeated for each of the authors. Since the relevant information is specified in variables such as formAuthor1 and formAuthor2, in the author loop first the values must be extracted from these variable (!) variable names. To this end, the PHP syntax ${'name'} is employed to determine the content of the specified variables in the nested parentheses:

// loop over all authors
$authCount=0;
for($i=1; $i<=$formAuthorsCount; $i++) {
  $newID = "none";
  // extract form variables
  $author = ${'formAuthor' . $i};
  $authorRadio = ${'formAuthorRadio' . $i};
  $authorType =${'formAuthorType' . $i};
  $author = trim($author);
  // author name in database-specific format
  $authorDB = last_name_first($author);

In the following lines, the ID number of the author is determined (variable newID). To accomplish this, the author is searched for in the database in both formats (given name followed by family name and conversely). If the name is not found, then it is inserted into the database as a new author with INSERT:

// author from text field
if($author and $authorType<3 or $authorRadio=="new") {
  // is the author already known to the database?
  $result =
    mysql_query("SELECT authID FROM authors " .
      "WHERE authName = " .  sql_str($authorDB) .
      " OR  authName = " .  sql_str($author));
  if(mysql_num_rows($result))
  $newID = mysql_result($result, 0, 0);
  // the author is unknown: insert into database
  else {
    mysql_query("INSERT INTO authors (authName) " .
    "VALUES('" . sql_str($authorDB) . ")");
    $newID = mysql_insert_id();
  }
  mysql_free_result($result);
}
// radio button with known author was chosen
elseif ($authorRadio!="new")
    $newID = $authorRadio;

The variable newID is inserted into the field authIDs, which contains all the authors of the new book title. Here it must be ensured that the same author is not inserted twice due to improper input:

// each author must be inserted only once into the authIDs
for($j=0; $j<sizeof($authIDs); $j++)
  if($authIDs[$j]==$newID) {
  $newID="none"; break;
  }
  // insert newID in authIDs
  if($newID!="none") {
      $authIDs[$authCount]=$newID;
      $authCount++;
  }
} //  end of the loop over author names

Only one INSERT command is required in order to enter the new book into the titles table, but because of the numerous columns, it is rather substantial. To create the command, the functions str_or_null and num_or_null are called on, which return the content of the respective variable or else NULL if the variable is empty:

// store book title
if($formCategory=="none") $formCategory = "";
if($formLanguage=="none") $formLanguage = "";
mysql_query("INSERT INTO titles (title, subtitle, " .
            "  edition, publID, catID, langID, year, " .
            "  isbn, comment) VALUES (" .
            sql_str(trim($formTitle)) . ", " .
            str_or_null($formSubtitle) . ", " .
            num_or_null($formEdition) . ", " .
            num_or_null($publID) . ", " .
            num_or_null($formCategory) . ", " .
            num_or_null($formLanguage) . ", " .
            num_or_null($formYear) . ", " .
            str_or_null($formISBN) . ", " .
            str_or_null($formComment) . ")");
$titleID = mysql_insert_id();

Finally, the relation between the book title and the authors must be created. For this, the INSERT command is assembled in the variable sql:

  // store entries in rel_title_author
  $sql = "INSERT INTO rel_title_author " .
      "(titleID, authID, authNr) VALUES ";
  for($i=0; $i<$authCount; $i++) {
    if($i!=0) $sql .= ",";
    $sql .= "($titleID, $authIDs[$i], $i)";
  }
  mysql_query($sql);
}

Room for Improvement

The script input.php does quite a bit, yet one can think of a number of ways in which it could be improved.

  • The program does not check whether the book title that is input perhaps already exists in the database. Such a control is not trivial to implement, because a comparison of titles alone is not sufficient. For example, there might certainly be more than one book with the title MySQL. Thus all additional information such as subtitle, publisher, edition, authors, all of which are subject to error, must be taken into consideration. A possible solution would be to display a list of all books with the same or similar title in the second input phase to lessen the danger of a duplicate being inserted. (If all titles were consistently accompanied by their ISBN numbers, then a test for uniqueness would be easy. But in our program such input is optional.)

  • In comparing author names, one can use the function SOUNDEX. This MySQL function returns the same value for similar-sounding English words (e.g., Green and Greene). In this way, possible typographical errors in author input can be repaired before it is too late. However, there is an assumption that the authors table is equipped with an additional soundex column in which all like-sounding names have been stored for existing authors. (It would be too costly to execute the SOUNDEX for all existing authors each time a comparison needs to be made.)

  • There should be better protection of save_data against errors. It is relatively unlikely, based on the prior validation, that a problem will occur in storing a name, but an error can be caused externally, for example, from a locking problem. In general, after every call to mysql_query a check against possible errors should be carried out.

  • A relatively obvious lack in the input form is that the number of authors is fixed in the first phase and cannot be changed in the second phase.

  • There is an inconsistency in that with input.php there is the possibility of storing new titles, authors, publishers, etc., but these data cannot be changed.

  • Data validation can be handled more easily with JavaScript. I am no great friend of JavaScript, preferring web pages that function without it. But no doubt, here at least one could well imagine a JavaScript option.

All of these goals are realizable, but they would make the code even more complex than it is already and begin to make this less of a demonstration example and more of areal-world application.


Managing the Book Categories (categories.php)

Each title in mylibrary can be associated with a category of books. This section is devoted to introducing a PHP page for managing these categories. With this page you will be able to create new categories and delete existing ones.

From the user's point of view, this management takes place over two pages: The initial page (see Figure 12-11) shows a hierarchical list of all the categories. You can select a category in order to insert subcategories (INSERT link) or delete one (DELETE link). If a category is deleted, then all of its subcategories are automatically deleted as well. Deletion is possible only if there is no book in the database that is associated with the category in question. Warning: Deletion takes place with no warning and no "undo" option.

Click To expand
Figure 12-11: Select or delete a category

The form for input of a new category appears in Figure 12-12. Only the part of the hierarchy that applies to the selected entry (shown in boldface) is displayed. It is possible to input several new subcategories at one time (separated by a semicolon).

Click To expand
Figure 12-12: Adding the category Visual Basic in the group All books | Computer books | Programming

Code Outline

The script categories.php begins with two auxiliary functions: insert_new_category stores a new category in the database; delete_category deletes a category and all of its subcategories. These two functions are described at the end of this section.

We now present the code for representing the hierarchical list of all categories as well as the representation and evaluation of a form for input of new categories. The following outline should provide orientation inside of categories.php:

<?php   file   mylibrary/categories.php
  // read in auxiliary functions
  include("private/mylibraryconnect.inc.php");
  // read form and URL variables
  $submitbutton  = array_item($_POST, 'submitbutton');
  $subcategories = array_item($_POST, 'subcategories');
  $catID         = array_item($_REQUEST, 'catID');
  $deleteID      = array_item($_REQUEST, 'deleteID');
  // ensure that magic quotes are present
  if(!get_magic_quotes_gpc())
      $subcategories = addslashes($subcategory);
  // define new functions
  function  insert_new_category($catID, $newcatName) {  . . .    }
  function  delete_category($catID) {  . . .    }
  // establish connection to database
  $connID = connect_to_mylibrary();
?>
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN">
<html><head>
  <meta http-equiv="Content-Type"
  content="text/html; charset=iso-8859-1" />
  <title>PHP Programming, categories of mylibrary database</title>
</head><body>
<?php
  // Main Code
  // if deleteID==n: delete category
  // if isempty(catID): display hierarchical list of all categories
  // else: - evaluate form data, insert new categories
  // - display hierarchical list of the selected category
  // - display form for input of a new category
php?>
</body></html>

URL Parameters

There are at most two variables that can be passed to the page as part of the address:

  • catID contains the ID number of the selected category (into which the new subcategory is to be inserted).

  • deleteID contains the ID number of the category that is to be deleted.

Form Variables

If the page used to evaluate form results is to be called, then two more variables are added to this list:

  • submitbutton contains "OK".

  • subcategories contains a character string with the new category to be added.

The rest of the code relies on the special characters in subcategories being prefixed by backslashes (magic quotes). To ensure that this is the case, addslashes is executed if get_magic_quotes_gpc returns the result false.

Main Code

The code for the PHP page begins with a test of both parameters that may have been passed: deleteID and catID. If deleteID is the ID number of exactly one record in categories, then delete_category is called to delete this category. (The test should ensure that no wild cards are smuggled in as deleteID so that more will be deleted than was intended.)

A similar test is carried out for catID. If this test is passed successfully, then the variable validCatID is set to TRUE.

// test whether deleteID refers to exactly one categories record
if($deleteID) {
  $result = mysql_query("SELECT catID FROM categories " .
  "WHERE catID='$deleteID'");
  // delete category
  if(mysql_num_rows($result)==1)
    delete_category($deleteID);
}
// test whether catID refers to an existing record
if($catID) {
  $result =
    mysql_query("SELECT catID FROM categories WHERE catID='$catID'");
  if(mysql_num_rows($result))
    $validCatID=TRUE;
}

Display Hierarchical List of All Categories

If catID is invalid or was not even passed, then a hierarchical list of all categories is displayed. For indenting the list the HTML tag <ul> is used. For each category the contents of the variable lastIndent is compared to the value of hierIndent from the database. If the indentation depth has changed, then the corresponding number of <ul> or </ul> tags are output.

For each category, a link to the insertion of a new record (with ?catID=n) or to the deletion of the category is displayed (with ?deleteID=n). These links are assembled via the auxiliary function build_href :

if(!$validCatID) {
  echo "<h2>Choose category</h2>\n";
  echo "<p>Click to insert/delete categories.</p>\n";
  // query over all categories
  $result = mysql_query("SELECT catName, catID, hierIndent " .
  "FROM categories ORDER BY hierNr");
  // evaluate query result
  echo "<p>\n";
  $lastIndent=-1;
  while($row=mysql_fetch_object($result)) {
    // <ul>'s for the next hierarchy level
    if($row->hierIndent > $lastIndent)
  echo "<ul>\n";
  // </ul>'s, to get out of the last valid hierarchy level
  if($row->hierIndent < $lastIndent)
  for($i=$row->hierIndent; $i<$lastIndent; $i++)
  echo "</ul>\n";
    $lastIndent = $row->hierIndent;
    // display category names as well as insert and delete links
  echo "<li>",
    htmlentities($row->catName), " (",
    build_href("categories.php", "catID=$row->catID", "insert"),
    ", ",
    build_href("categories.php", "deleteID=$row->catID",
    "delete"),
    ")",
    "</li>\n";
  }
  // close all <ul>'s still open
  for($i=-1; $i<=lastIndent; $i++)
      echo "</ul>\n";
  echo "</p>\n";
}

Inserting New Categories

The else that begins here runs almost to the end of the PHP file. It is not the entire category list that should be displayed, but only one particular category into which new subcategories may be added (see Figure 12-12).

However, this code segment is executed only if form results are to be evaluated. To accomplish this, the variable subcategories is decomposed into an array containing all the new categories to be inserted. They are inserted into the database with insert_new_category. The variable count keeps track of how often this is successfully accomplished. A brief message reports on the result:

else {// to if(!validCatID)
    echo "<h2>Insert new categories</h2>\n";
    // evaluate form data (insert new subcategories)
    if($subcategories){
        $subcatarray = explode(";", $subcategories);
        $count=0;
        while($i = each($subcatarray)) {
    $newcatname = trim($i[1]);
    $count += insert_new_category($catID, $newcatname);
        }
        if($count)
    echo "<p>$count new categories have been inserted ",
    "into the categories table.\n";
    }

Representing a Branch of the Hierarchical Tree

The following lines of code have the purpose of displaying the category described by catID together with its super- and subcategories (see Figure 12-12). The purpose, then, is to display a small segment of the entire hierarchical tree.

In the first step, three arrays (catNames, catIDs, parentCatIDs) are constructed, which contain the higher levels of the hierarchy (catNames[0], i.e., all books; catNames[1], e.g., Computer books; catNames[2], e.g., Programming, etc.); see Figure 12-12. The arrays are then filled with the data of the current record described by catID. Thereafter, a loop follows parentCatIDs back to the root of the hierarchy.

After these data have been determined, they are displayed in a loop (indented as required by means of <ul>). The current category is treated in the same manner, except that it is displayed in boldface type:

// determine data of the current category (catID)
$result =
    mysql_query("SELECT * FROM categories WHERE catID='$catID'");
$row = mysql_fetch_object($result);
$maxIndent = $row->hierIndent;
$catNames[$maxIndent] = $row->catName;
$catIDs[$maxIndent] = $row->catID;
$parentCatIDs[$maxIndent] = $row->parentCatID;
// loop to determine the data of the next higher category
for($i=$maxIndent-1; $i>=0; $i--) {
  $result = mysql_query("SELECT * FROM categories " .
    "WHERE catID='" . $parentCatIDs[$i+1] . "'");
  $row = mysql_fetch_object($result);
  $catNames[$i] = $row->catName;
  $catIDs[$i] = $row->catID;
  $parentCatIDs[$i] = $row->parentCatID;
}
// display next higher category
for($i=0; $i<$maxIndent; $i++) {
    echo "<ul><li>", htmlentities($catNames[$i]), "</li>\n";
}
// display current category
echo "<ul><li><b>", htmlentities($catNames[$maxIndent]),
    "</b></li>\n";

To give a view as to which subcategories already exist, all existing subcategories of catID are determined and displayed. The categories are supplied with a DELETE link, so that incorrectly inserted categories can be deleted:

// display all subcategories of catID together with a delete link
$result =
    mysql_query("SELECT catName, catID FROM categories " .
            "WHERE parentCatID='$catID' ORDER BY catName");
echo "<ul>\n";
if(mysql_num_rows($result)) {
  while($row=mysql_fetch_object($result)) {
  echo "<li>", htmlentities($row->catName), " (",
  build_href("categories.php",
  "catID=$catID&deleteID=$row->catID",
  "delete") .
  ")</li>\n";
  }}
else {
  echo "(No subcategories yet.)\n";
}
echo "</ul>\n";

Finally, all open <ul> tags must be closed:

// close hierarchical category list
for($i=0; $i<=$maxIndent; $i++) {
    echo "</ul>\n";
}

Input Form for New Categories

The input form for new categories consists merely of a text field (subcategories) anda "submit" button. The action link refers again to categories.php, where additionally, the current catID number is transmitted. This has the effect that as a result of the input, the reduced hierarchical tree and the form are shown. Thus the result of the input is immediately visible (in the form of additional subcategories). Moreover, additional categories can be immediately input:

  echo '<form method="post" action="categories.php?catID="',
    $catID, '">', "\n";
  echo "<p>Insert new sub-categories to ",
    "<b>$catNames[$maxIndent]</b>. <br />You may add several ",
    "subcategories at once. <br />Use ; to seperate ",
    "your entries.</p>\n";
  echo '<p><input name="subcategories" size="60" maxlength="80" />',
    "\n";
  echo '<input type="submit" value="OK" name="submitbutton" /></p>',
    "\n";
echo "</form>\n";

The PHP code ends with a link that leads back to the complete category list. The closing curly brace ends the long else block following if(!validCatID):

  // link back to complete category list
  echo "<p>Back to full ",
    build_href("./categories.php", "", "categories list") . ".\n";
}

Function to Save a New Category

The function insert_new_category adds a new category to the table categories. The ID number (catID) of the category to which the new category (newcatName) is to be subordinated must be passed as parameter.

The function assumes that catID is valid. With newcatName, a test is made as to whether the character string is empty or whether perhaps the new category already exists as a subcategory of catID. If either case obtains, the function is immediately terminated:

function  insert_new_category($catID, $newcatName) {
  // $catID: category into which the new subcategory is to be inserted
  // $newcatName: name of the new subcategory
  // terminate if newcatName is empty
  if(!$newcatName) return(0);
  // test whether the category already exists
  $result = mysql_query("SELECT catID FROM categories " .
    "WHERE parentCatID=$catID " .
    "  AND catName='$newcatName'");
  if(mysql_num_rows($result)) return(0);
  mysql_free_result($result);

Now comes the actualization of the insertion algorithm that was explained in detail in Chapter 5. Briefly, the position of the new category within the hierarchical category list is found (that is, the hierNr value for the new category).

Note that LOCK TABLE categories WRITE should be executed before SELECT, INSERT, and UPDATE commands. This ensures that during a change in the categories table, no other program can make changes to that table. (Of course, that is an extremely unlikely scenario in this example, but it is theoretically possible.) At the end of the algorithm, the blocking must be lifted with UNLOCK TABLES. The LOCK/UNLOCK commands have been commented out in the code, because many database users do not have the lock table privilege in MySQL by default. Change the code and experiment for yourself.

Tip

If you are working with InnoDB tables, it is more efficient to execute the entire algorithm as a transaction, rather than using LOCK/UNLOCK commands. Instead of LOCK, execute SET AUTOCOMMIT = 0 and BEGIN, and instead of UNLOCK, execute COMMIT and SET AUTOCOMMIT=1.

// protect table categories against external changes
// mysql_query_test("LOCK TABLE categories WRITE");
// (1) get information on the higher-ranking category
$result =
    mysql_query("SELECT * FROM categories WHERE catID='$catID'");
$base=mysql_fetch_object($result);
mysql_free_result($result);
// (2a) search for the category below $newcatName
$result =
  mysql_query("SELECT hierNr FROM categories " .
    "WHERE parentCatID='$base->catID' " .
    "" AND catName>'$newcatName' " .
    "ORDER BY catName LIMIT 1");
if(mysql_num_rows($result)) {
    $newhierNr = mysql_result($result, 0, 0);
    mysql_free_result($result);
}
// (2b) search for the start of the next category group
else {
  $result =
    mysql_query("SELECT hierNr FROM categories " .
    "WHERE hierNr>'$base->hierNr' " .
    "AND hierIndent<='$base->hierIndent' " .
    "ORDER BY hierNr LIMIT 1");
if(mysql_num_rows($result)) {
    $newhierNr = mysql_result($result, 0, 0);
    mysql_free_result($result);
  }
  // (2c) search for the maximal hierNr in the table
  else {
    $result =
    mysql_query("SELECT MAX(hierNr) FROM categories");
    $newhierNr = mysql_result($result, 0, 0)+1;
    mysql_free_result($result);
  }
}

Once hierNr has been determined, this value is increased by 1 for all existing records with the same or higher hierNr, so as to make room for the new record. Then the new record is inserted:

// (3) increase hierNr for all categories below $newcatName
mysql_query("UPDATE categories SET hierNr=hierNr+1 " .
  "WHERE hiernr>=$newhierNr");
// (4) insert new category
mysql_query("INSERT INTO categories " .
  "  (catName, parentCatID, hierNr, hierIndent) " .
  "VALUES ('$newcatName', $catID, $newhierNr, " .
   ($base->hierIndent+1) . ")");
// remove the LOCK
// mysql_query_test("UNLOCK TABLES");
return(1); }
Remark

To save on resources, in this function and the following to be introduced, delete_category, allSQL query results will be released with mysql_free_result. This makes sense because within the function a number of queries are executed, and because the function can be called a number of times during the lifetime of the script (for example, if several new categories are to be added at one time).

A Function to Delete an Existing Category

As parameter to the function delete_category the ID number of the category to be deleted is passed. Within the function it is determined whether there are any subcategories to this category. If that is the case, then delete_category is called recursively to delete these subcategories. During this process a running total of return values is calculated, where 1 indicates that the category was able to be deleted, and 0 that it could not be. This allows a simple determination to be made as to whether all the subcategories were deleted. If that is not the case, then the function is terminated:

function  delete_category($catID) {
  // search for subcategories of catID
  // and delete them with recursive calls to delete_category
  $result =
  mysql_query("SELECT catID FROM categories " .
  "WHERE parentCatID='$catID'");
  $rows = mysql_num_rows($result);
  $deletedRows = 0;
  while($row=mysql_fetch_row($result)) {
    $deletedRows += delete_category($row[0]);
  }
  mysql_free_result($result);
  // if not all subcategories could be deleted,
  // then the category catID cannot be deleted
  if($deletedRows != $rows)
    return(0);

Then a test is made as to whether there are book titles that belong to the category that is to be deleted. In such a case, the category cannot be deleted. If there are such titles, an error message is output that gives the name of the category and the number of associated titles.

// are there titles that belong to catID?
$result = mysql_query("SELECT COUNT(titleID) FROM titles " .
    "WHERE catID='$catID'");
// yes, the category may not be deleted;
// display error message
if($nrOfTitles = mysql_result($result, 0, 0)) {
  mysql_free_result($result);
  $result = mysql_query("SELECT catName FROM categories " .
      "WHERE catID='$catID'");
  $catName = mysql_result($result, 0, 0);
  mysql_free_result($result);
  echo "<br />Category <b>$catName</b> is used in $nrOfTitles ",
    "titles. You cannot delete it and its parents.\n";
  return(0);
}

Otherwise, there is nothing to stop the hand of destruction: The category can now be deleted.

  // delete category
  mysql_query("DELETE FROM categories WHERE catID='$catID'");
  return(1);
}
Remark

Deleting categories can result in holes in the sequence of hierNr values in the categories table. Thus you may not assume that the difference of two hierNr values represents the number of categories of intermediate records.


Ideas for Improvements and Extensions

The three PHP pages displayed do not, unfortunately, represent a complete user interface to the mylibrary database. The following list presents some of the most pressing improvements that might be made:

  • Altering existing records (for example, to correct typographical errors in title, author, or publisher names; change the association title « publisher or title « category).

  • Delete existing records (title, author, publisher, etc.), with queries that ensure that no empty cross references remain.

  • Unification of duplicates (for example, if an author has been stored two or three times due to a typographical error).

  • Management of the table languages.

None of these points represents a difficult problem, but the amount of programming required is rather large. To keep the code readable, the various auxiliary functions should be organized on a modular plan.

However, one may ask the question whether such an HTML-based user interface is worth the effort. If the user interface were to be developed with a freestanding programming language (Java, Visual Basic, Delphi, for example), then the amount of work would not be much greater, but one could achieve greater ease of use and perhaps even greater speed.


Chapter 13: PHP - Discussion Forum

Team LiB

Chapter 13: PHP — Discussion Forum

Overview

The example of this section shows how a discussion forum managed by its users can be realized with several PHP script files and the database myforum introduced in Chapter 5.

Pointer

As with all the examples in this book, we are interested more in explicating programming techniques than in producing a polished application. If your ambitions range in the direction of creating a web discussion forum, it would be worthwhile looking into the numerous off-the-shelf solutions available. The two currently most popular PHP scripts for discussion forums (or "fora" for all you Latinate purists out there) can be found at the following addresses:


How to Run It

From the user's viewpoint, the portals leading to the discussion forum are opened by a login dialog (see Figure 13-1).

Click To expand
Figure 13-1: Login dialog for the myforum discussion forum

One need not log in if one wishes only to read the postings. For this purpose a direct link to all forums is offered. A login is, however, required for those wishing to post messages. But first it is necessary to establish a myforum account (see Figure 13-2).

Click To expand
Figure 13-2: Establishing a new myforum account

(If you would like to try out the example on my web site, you can save yourself the trouble of establishing an account by logging in under the name test with the password secret.)

With or without registration you arrive at the page forumlist.php, in which all discussion forums are listed for your perusal (each with the language of the forum indicated). At the start of the page is displayed the user name of the current login (or anonymous). See Figure 13-3.

Click To expand
Figure 13-3: List of all discussion forums

On the page forumread.php (see Figure 13-4) the last fifty messages to have been posted are displayed. The most current discussion threads are displayed at the start of the page. Within the thread appears the place in the sequence in which each article was posted (oldest article first). To make clear which contributions belong to which threads, the background color changes for each thread. If there are more than fifty articles, then at the end of the page there appear links to the older contributions.

Click To expand
Figure 13-4: List of all postings to testforum

Individual messages are displayed by means of the script forummessage.php (see Figure 13-5). If there are several replies to a given message, then the entire thread (a list of all messages together with their texts) can be displayed on a single page with forumthread.php (see Figure 13-6).

Click To expand
Figure 13-5: A single posting
Click To expand
Figure 13-6: An entire discussion thread

The script forumwrite.php (see Figure 13-7) assists in composing a new message or responding to an existing one.

Click To expand
Figure 13-7: Form for composing a message


Overview of the Program Code

Our example consists of the following files:

myforumconnect.inc.php | auxiliary functions (including database login)
myforumpassword.inc.php | login information and password
forumlogin.php | login to the forum
forumnewlogin.php | new registration
forumlist.php | display list of all discussion groups
forumread.php | display list of titles of all messages
forumthread.php | display discussion thread
forummessage.php | display discussion message
forumwrite.php | compose message
forumlogout.php | unregister
Remark

The example files assume at least PHP 4.1. A version of the entire example compatible with PHP 4.0 can be found at my web site (http://www.kofler.cc) among the PHP example files for the first edition of this book.

The File myforumconnect.inc.php

The file myforumconnect.inc.php contains several auxiliary functions. The most important of these is connect_to_myforum, which establishes the connection to MySQL. These functions are nearly identical to those of mylibraryconnect.inc.php (see Chapter 12), and so we will not describe them here.

Managing a Session

Pointer

If you have never worked with PHP sessions, you should do some reading in this important area. Every book on PHP deals with this topic (in some, exhaustively to the point of exhaustion). The problem with many PHP books, however, is that they consider only PHP 4.0. Much has changed since then. Therefore, have a look at the official PHP documentation at the following address:

http://www.php.net/manual/en/ref.session.php

Passing information about who has registered between the script files is managed by means of a PHP session (variables sesUserID and sesUserName). Otherwise, the passing of information between the script files is carried out with URL parameters.

Since the required code for managing the session variables is in more or less the same form in all the script files, it is described here in one place. At the beginning of each script the function session_start is executed in order to activate the session support. Access to session variables is now accomplished through $_SESSION['name'].The following lines provide an example:

session_start();
if(empty($_SESSION['sesUserID'])) {
    $_SESSION['sesUserName'] = "anonymous";
}

With many web browsers there is a problem in connection with sessions: In the attempt to return to an already visited form using the BACK button, all input to the form is deleted. At times, this can increase the security of the application, but often, it is simply a pain in the neck for the user if there are corrections to be made in the previous input. If you want the BACK button to function as it should, then you should execute the following line immediately after session_start:

header("Cache-control: private"); // allow backspace in forms

Sessions Without Cookies

If the client permits cookies (that is, in the user's browser), then all of this happens automatically; that is, the content of the session variables continues to be available when a switch is made to a new page.

On the other hand, if cookies are not allowed, then a session identification number (SID) must be passed as part of the URL at each page change. If the PHP interpreter is appropriately compiled (option --enable-trans-sid) and configured (variable session.use_trans_sid in php.ini), then the SID will be automatically built into every URL. But do not rely on this! This automation is deactivated by default.

In order to ensure that management of the session functions dependably in all circumstances, the evaluation of the constant SID and a bit of hand operation are necessary. The constant SID contains a character string of the form PHPSESSID=3ff5... . This character string must be specified as a URL parameter at each change between PHP scripts. To avoid constant queries of the form if(SID) ... , the two variables sid1 and sid2 are defined on the basis of SID.

if(SID) {
    $sid1 = "?" . SID;
    $sid2 = "&" . SID; }

In what follows, sid1 (first URL parameter) or sid2 (additional URL parameter) can simply be attached to URL character strings, for example, as follows:

// page change to forumlist.php
header("Location: ./forumlist.php$sid1");
// write link to page forumwrite.php in the HTML document
$query = "forumID=$row->forumID&msgID=$row->msgID" . $sid2;
echo "<p>",
    build_href("./forumwrite.php", $query, "Reply to this message"),
    ".</p>\n";


Registration (forumlogin.php)

The script forumlogin.php displays a simple form in which a login name and password can be given (see Figure 13-1). The evaluation of the form variables formUser and formPassword also takes place in this script:

<?php // php/myforum/forumlogin.php
// read in standard functions
include("myforumconnect.inc.php");
// Form variables
$formSubmit       = array_item($_POST, 'formSubmit');
$formPassword     = array_item($_POST, 'formPassword');
  $formUser       = array_item($_POST, 'formUser');
// ensure that special characters are marked by \
if(!get_magic_quotes_gpc()) {
    $formPassword = addslashes($formPassword);
    $formUser         = addslashes($formUser); }
// session management
session_start();
header("Cache-control: private"); // backspace in forms
$_SESSION['sesUserID']     = '';       // delete any existing
$_SESSION['sesUserName'] = '';       // data
// sid1/-2 help in generating URLs to other pages
// if the user prohibits cookies
if(SID) {
    $sid1 = "?" . SID;
    $sid2 = "&" . SID; }
else {
    $sid1=""; $sid2=""; }
// create connection to the database
$connID = connect_to_myforum();

If the users database contains the character string pair specified in the form (username and password), then the sessions variables sesUserID and sesUserName are initialized. Then the PHP function header is used to effect a change to the page forumlist.php. (In order for header to work, no HTML output can occur before this function. For this reason, the HTML header is output relatively late in forumlogin.php.)

// evaluate form data
if($formSubmit) {
  $result =
    mysql_query("SELECT userID, userName FROM users " .
     "WHERE userName = '$formUser' " .
     "  AND password = PASSWORD('$formPassword')");
  // ok, user recognized
  if(mysql_num_rows($result)==1) {
    $_SESSION['sesUserID']     = mysql_result($result, 0, 0);
    $_SESSION['sesUserName'] = mysql_result($result, 0, 1);
    // change to page forumlist.php
    header("Location: ./forumlist.php$sid1");
    exit;
  }
}
?>

The remainder of the code serves to display the input form and some links and additional information:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN">
<html><head>
  <meta http-equiv="Content-Type"
  content="text/html; charset=iso-8859-1" />
<title>PHP Programming, forum login (myforum database)</title>
</head><body>
<h2>Login to the <i>myforum</i> discussion forum</h2>
<?php
// error message if the login attempt fails
if($formSubmit and empty($userID))
    echo '<p><font color="#ff0000">Username or password ',
      "are invalid. Please try again.</font></p>\n";
?>
<p>  . . .   some text  . . .    </p>
<form method="post" action="forumlogin.php">
<p><table>
<tr><td>User name:</td>
      <td><input name="formUser" size="30" maxlength="30" /></td>
<tr><td>Password:</td>
      <td><input type="password" name="formPassword" size="30" maxlength="30" /></td>
<tr><td></td>
      <td><input type="submit" value="Login" name="formSubmit" /></td>
</table></p>
</form>
</body></html>


Defining a New User (forumnewlogin.php)

A new user for myforum can be defined in the form forumnewlogin.php (see Figure 13-2). A user name, e-mail address, and (twice to avoid typos) password are required:

<?php // php/myforum/forumnewlogin.php
// initialization code similar to that in forumlogin.php
// Form variables
$formSubmit        = array_item($_POST, 'formSubmit');
$formPassword1     = array_item($_POST, 'formPassword1');
$formPassword2     = array_item($_POST, 'formPassword2');
    $formUser      = array_item($_POST, 'formUser');
$formEMail         = array_item($_POST, 'formEMail');
// no magic quotes
if(get_magic_quotes_gpc()) {
  $formPassword1 = stripslashes($formPassword1);
  $formPassword2 = stripslashes($formPassword2);
    $formUser  = stripslashes($formUser);
  $formEMail     = stripslashes($formEMail); }

The actual program code begins with the evaluation of the form variables formUser, formEMail, formPassword1, and formPassword2. The first thing to occur is a check as to whether the name is already in use. If that is the case, then the variable wrongName is set to 1 (for the purpose of a later error message to be displayed). Additionally, a test is made as to whether an e-mail address was given (without verifying its format) and whether the two passwords are identical:

// evaluate form variables
if($formSubmit=="OK") {
  $formUser = trim($formUser);
  $formEMail = trim($formEMail);
  $formPassword1 = trim($formPassword1);
  $formPassword2 = trim($formPassword2);
  // is the user name available?
  if($formUser) {
      $result =
  mysql_query("SELECT userID, userName FROM users " .
      "WHERE userName = '" . addslashes($formUser) . "' ");
      if(mysql_num_rows($result)==1)
  $wrongName = 1; // name is in use
  }
else
  $wrongName = 1;// no name was given whatsoever
// was an e-mail address given?
if(!$formEMail)
  $wrongEMail=1;
// do the passwords agree?
if($formPassword1=="" or $formPassword1!=$formPassword2)
    $wrongPassword=1;

If no formal errors are detected, then the new user is stored in the user database with an INSERT. Then the session variables sesUserID and sesUserName are initialized and the page forumlist.php displayed:

  // ok?
  if(!($wrongName || $wrongPassword || $wrongEMail)) {
    // store user name and password, determine ID,
    mysql_query("INSERT INTO users (userName, email, password) " .
    "VALUES ('" . addslashes($formUser) . "', '" .
     addslashes($formEMail) . "', " .
  "        PASSWORD('" . addslashes($formPassword1) .
  "'))");
    $sesUserID = mysql_insert_id();
    $sesUserName = stripslashes($formUser);
    // page change to forumlist.php
    header("Location: ./forumlist.php$sid1");
    exit;
  }
}
?>

The remaining code serves to display the form and any error messages that might be required:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN">
<html><head>
  <meta http-equiv="Content-Type"
  content="text/html; charset=iso-8859-1" />
<title>PHP Programming, Get a new account (myforum database)</title>
</head><body>
<h2>Get a new account for the <i>myforum</i> discussion forum</h2>
<p>Please specify your user name  . . .
<?php
// display error messages
if($wrongName) echo "  . . .    ";
    if($wrongEMail) echo "  . . .";
    if($wrongPassword) echo " . . . ";
?>
<form method="post" action="forumnewlogin.php<?php echo $sid1;?>">
<p><table>
<tr><td>User name:</td>
    <td><input name="formUser" size="30" maxlength="30"
      value="<?php echo htmlspecialchars($formUser); ?>" /></td>
<tr><td>Email address:</td>
    <td><input name="formEMail" size="30" maxlength="30"
      value="<?php echo htmlspecialchars($formEMail); ?>" /></td>
<tr><td>Password: </td>
    <td><input type="password" name="formPassword1"
      size="30" maxlength="30"
      value="<?php echo htmlspecialchars($formPassword1); ?>" />
    </td>
<tr><td>Password (again): </td>
    <td><input type="password" name="formPassword2"
      size="30" maxlength="30"
      value="<?php echo htmlspecialchars($formPassword2); ?>" />
    </td>
<tr><td></td>
    <td><input type="submit" value="OK" name="formSubmit" /></td>
</table></p>
</form> </body></html>

Possibilities for Expansion

Verifying an e-mail address: The easiest improvement to implement is to send an initial access code to the specified e-mail address (optimally in the form of an HTTP link). You could use a random number, for example, that is generated at the time of registration and stored in the user table until the user first accesses the account.


Display List of All Discussion Groups (forumlist.php)

The script forumlist.php displays a list of all discussion groups defined in the table forums (see Figure 13-3). The list is determined by means of a simple SELECT instruction and represented in the form of links that refer to the script forumread.php:

<?php // php/myforum/forumlist.php
include("myforumconnect.inc.php");
session_start();
if(empty($_SESSION['sesUserID'])) {
    $_SESSION['sesUserName'] = "anonymous"; }
if(SID) {  . . .    }     //  code as in forumlogin.php
?>
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN">
<html><head>
    <meta http-equiv="Content-Type"
      content="text/html; charset=iso-8859-1" />
<title>PHP Programming, list of discussion groups
     (myforum database)</title>
</head><body>
<h2>List of <i>myforum</i> discussion forums</h2>
<?php
echo "<p>Username: ", htmlentities($_SESSION['sesUserName']),
"</p>\n";
// display list of all discussion groups
echo "<p>List of groups:</p>\n";
echo "<ul>\n";
$result =
    mysql_query("SELECT forumID, forumName, language FROM forums " .
        "ORDER BY forumName, language ");
while($row=mysql_fetch_object($result)) {
  echo "<li>",
    build_href("forumread.php",
      "forumID=$row->forumID" . $sid2,
      "$row->forumName ($row ->language)"), "\n"; }
echo "</ul>\n";
// link to the logout script
echo "<p><a href=\"forumlogout.php$sid1\">Logout</a>\n";
?>
</body></html>


Message List of a Discussion Group (forumread.php)

From the viewpoint of database programming, the script forumread.php must be considered one of the most interesting modules of this example. Its task consists in displaying the fifty most recent messages of a discussion group (see Figure 13-4). If the discussion group contains more than that number of messages, then forumread.php enables the user to leaf pagewise through the messages.

Various Representations of Hierarchical Lists

The problem in representing the list of messages is that the pagewise division of the list results in the separation of discussion threads. (A recent message can arise as the reaction to a much older thread.)

To clarify the situation, we present Figure 13-8, which shows three discussion threads with seventeen messages altogether from users A through G. The threads are ordered chronologically (the newest thread first), and within the threads the messages are ordered hierarchically based on who was responding to whom.

Many discussion forums simply do not display the message hierarchy with the pagewise display of messages. The advantage of this is that such a solution is easy to implement, and the messages can be ordered according to their creation dates. However, the flow of the discussion is lost (see Figure 13-9 with the ten most current messages from Figure 13-8 ordered by creation time).

In Figure 13-10, on the other hand, an attempt is made to retain the hierarchy. In the most current thread this effort has brought success. In the two older threads the messages are properly indented, but the flow of the discussion is incomplete due to missing messages. The order of messages still corresponds to the thread structure, but the result is confusing.

Figure 13-8: Hierarchical representation of three threads (ordered by flow of the discussion)

Figure 13-9: Flat representation of the ten most recent messages from Figure 13-8 (ordered by time ofcreation)

Nonetheless, we have chosen this solution for forumread.php, because it seemed a reasonable compromise. (The number of messages per page is 50, and this number can be reset if it is so desired. Of course, the larger this value, the less often will there be dangling threads.) For threads whose starting message cannot be completely displayed on a page, the link (show entire thread) is displayed, which leads to the script forumthread.php. There the entire thread is displayed regardless of its length.

Figure 13-10: Hierarchical representation of the most recent ten messages (ordered according to the discussion flow)
Remark

Some discussion forums offer the possibility of expanding and contracting threads dynamically. This would be the most elegant solution, of course. However, in practice, the entire page must be reloaded every time the level of the representation is changed, which leads to annoying delays.

Another solution consists in displaying on a page only the title of the first message, as a thread overview, so to speak. Only when the thread title is clicked on does a list appear with all the messages in this thread. I have chosen this method for the discussion forum on my web site www.kofler.cc. This variantis easy to read, but at first glance, it is difficult to see which messages arrived most recently and which threads are likely to be of interest.

There is probably no truly perfect solution for HTML-based discussion forums. Forums that are very active are better off being organized as news groups, for which there are programs for reading threads.

Program Structure

URL Variables

Two URL variables can be passed to forumread.php:

  • forumID determines the forum whose messages are to be displayed.

  • page specifies the page that is to be displayed.

Program Code

The program code begins with the by now well-known initialization instructions. If forumID is empty, the script is terminated, and instead, forumlist.php is displayed:

<?php // php/myforum/\{forumread.php}
// read in standard functions,
include("myforumconnect.inc.php");
// read URL variables
$forumID = array_item($_REQUEST, 'forumID');
$page       = array_item($_REQUEST, 'page');
// session management
session_start();
if(empty($_SESSION['sesUserID'])) {
    $_SESSION['sesUserName'] = "anonymous"; }
if(SID) {  . . .    }     //  Code wie in forumlogin.php
if(empty($forumID)) {
    header("Location: forumlist.php$sid1");
    exit; }
?>

Validating ForumID

The following lines contain the HTML header. Then a SELECT query checks whether forumID refers to an existing discussion forum. In the SELECT query, LIMIT is used to ensure that no wild cards are passed in forumID. If forumID is valid, then the name of the forum is displayed:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN">
<html><head>
    <meta http-equiv="Content-Type"
        content="text/html; charset=iso-8859-1" />
<title>PHP Programming, browse discussion group (myforum database)</title>
</head><body>
<h2>Browse discussion group</h2>
<?php
echo "<p>Username: ", htmlentities($_SESSION['sesUserName']), "</p>\n";
// links forumID to a discussion group
$forumID = trim($forumID);
$result = mysql_query("SELECT forumName, language FROM forums " .
      "WHERE
if(!$result or mysql_num_rows($result)!=1) {
  echo '<p><font color="#ff0000">ID number of forum seems ',
    "to be invalid.</font> Please choose a forum in the ",
    "<a href=\"forumlist.php$sid1\">forum list</a>.</p>\n";
  echo "</body></html>\n";
  exit;
}
// display name of the forum
$row = mysql_fetch_object($result);
echo "<h3>Forum ", htmlentities($row->forumName),
    "  (", htmlentities($row->language),     ")</h3>\n";
mysql_free_result($result);

Determining the Message Text

The page size is determined by the variable with the unsurprising name pagesize. If a valid page number was not specified, then the page number is set to 1:

$pagesize = 50;
if(!$page or !(is_numeric($page)) or $page<0 or $page>100)
  $page=1;

In the following, the list of message titles to be displayed is determined. To effect this, first the temporary table tmpMsgIDs with the msgID numbers of the desired page is generated. The variable moremessages is later evaluated to build into the HTML page a link to the next page with additional messages:

mysql_query("DROP TABLE IF EXISTS tmpMsgIDs");
mysql_query("CREATE TEMPORARY TABLE tmpMsgIDs TYPE = HEAP " .
    "SELECT msgID FROM messages " .
    "WHERE forumID = $forumID " .
    "ORDER BY timest DESC " .
    "LIMIT " .  ($pagesize*($page-1)) . ", " .  ($pagesize+1));
if(mysql_affected_rows() > $pagesize)
$moremessages=1;

In a second query, the data msgID, subject, level, rootID, timest, and username are determined for all messages contained in tmpMsgIDs. What is rather unconventional is the sorting of the messages: With ORDER BY LEFT(orderstr, 4) DESC we achieve that the creation time of the thread is used as the first sort criterion. New threads are displayed at the head of the list. As a second sort criterion, orderstr is used. This results in the messages being sorted hierarchically based on the flow of the discussion (see also Chapter 5, where the design of the myforum database is discussed):

$result =
  mysql_query("SELECT messages.msgID, subject, level, rootID, " .
    "  DATE_FORMAT(timest, '%Y/%c/%e %k:%i') AS timest, " .
    "  username " .
    "FROM messages, users, tmpMsgIDs " .
    "WHERE messages.msgID = tmpMsgIDs.msgID " .
    "  AND messages.userID = users.userID " .
    "ORDER BY LEFT(orderstr, 4) DESC, orderstr");
// delete temporary table
mysql_query("DROP TABLE tmpMsgIDs");

Display Message List

The following loop displays the results of the above query. Each message is represented as an HTML link on the page forummessage.php. If the first message of the thread cannot be displayed, then instead, a link to forumthread.php is inserted, so that the entire thread can be displayed. This special case is shown in Figure 13-11 (message 4).

Click To expand
Figure 13-11: Representation of messages via forumread.php

The threads are indicated alternately with two different shades of gray, so that it is immediately clear which message belongs to which thread. For this, the variable threadcounter in increased by 1 at the start of each new thread. According to whether threadcounter is even or odd (test using the modulo operator %), the variable color is initialized with a lighter or darker shade of gray:

}// display list with subject text, author name, and date
$oldthreadID = 0;
$threadCounter = 0;
if(mysql_num_rows($result)) {
    echo '<table border="0" cellpadding="5" cellspacing="0">', "\n";
while($row=mysql_fetch_object($result)) {
    // change background for a new thread
    if($row->rootID!=$oldthreadID) $threadCounter++;
    if($threadCounter % 2)
  $color = '"#eeeeee"';
    else
  $color = '"#dddddd"';
    // display link to entire thread if there are no messages
    if($row->rootID!=0 and $row->rootID!=$oldthreadID) {
  $query = "rootID=$row->rootID" . $sid2;
  echo "<tr bgcolor=$color><td>",
      build_href("forumthread.php", $query, "(show entire thread)"),
      "</td><td></td><td></td>\n";
    }
    // display subject, name, and date
    $query = "msgID=$row->msgID" . $sid2;
    $shortSubj = substr($row->subject, 0, 55);
    if(strlen($row->subject)>55)
  $shortSubj .= "  . . .    ";
      echo "<tr bgcolor=$color><td>",
  $row->level ?
      str_repeat("&nbsp;&nbsp;&nbsp;", $row->level) : "",
  build_href("forummessage.php", $query, $shortSubj), "</td>",
  "<td align=\"center\"><i>", htmlentities($row->username),
  "</i>", "</td>",
  "<td align=\"right\"> $row->timest </td>\n";
    // update oldthreadID (to recognize new threads)
    if($row->rootID==0)
  $oldthreadID=$row->msgID;
    else
  $oldthreadID=$row->rootID;
  }
  echo "</table>\n";
}

Links to Other Pages

The program code ends with the output of links to later and earlier messages and to the form for input of a new message:

echo "<p><hr />\n";
// Are there new mesages?
if($page>1) {
  $query = "forumID=$forumID&page=" . ($page-1) . $sid2;
  echo build_href("forumread.php", $query, "Show newer messages.");
  if($moremessages) echo " / ";
}
// Are there older messages?
if($moremessages) {
  $query = "forumID=$forumID&page=" . ($page+1) . $sid2;
  echo build_href("forumread.php", $query, "Show older messages.");
}
echo "</p>\n";
// link to write a new message
if($page>1 or $moremessages)
  echo " / ";
$query = "forumID=$forumID" . $sid2;
echo build_href("forumwrite.php", $query, "Write a new message"),
  ".\n";
// links to the list of all forums
echo "<p>", build_href("forumlist.php", SID, "List of all forums"), ".</p>\n"; ?>
</body></html>


Representation of Message Text (forummessage.php)

In forumread.php the title of each message is linked to forummessage.php. This script displays not only the message text, but also links for creating a reply and for the display of the complete thread (see Figure 13-5). The message to be displayed is passed to the script via the URL variable msgID.

Once again, the program code begins with our tried and true initialization. If msgID is empty, then the script is terminated, and instead, forumlist.php is displayed:

<?php // php/myforum/forummessage.php
// read URL variable
$msgID = array_item($_REQUEST, 'msgID');
// the rest of the initialization is shown in the other
// forum examples
// if $msgID empty, display forumlist.php
if(empty($msgID)) {
    header("Location: ./forumlist.php$sid1");
exit; }
?>
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN">
<html><head>
  <meta http-equiv="Content-Type"
  content="text/html; charset=iso-8859-1" />
<title>PHP Programming, show message (myforum database)</title>
</head><body>
<h2>Show myforum message</h2>

The heart of the code begins with a test as to whether msgID actually refers to a message. If the SELECT is successful, then all the data are retrieved that will need to be displayed (subject, msgText, etc.):

<?php
$msgID = trim($msgID);
$result =
  mysql_query("SELECT msgID, rootID, subject, msgText, " .
    "  username, forumName, messages.forumID, " .
    "  DATE_FORMAT(timest, '%Y/%c/%e %k:%i') AS timest " .
    "FROM messages, users, forums " .
    "WHERE msgID='$msgID' " .
    "  AND messages.userID = users.userID " .
    "  AND messages.forumID = forums.forumID " .
    "LIMIT 2");
if(!$result or mysql_num_rows($result)!=1) {
  echo '<p><font color="#ff0000">ID number of message seems ',
    "to be invalid.</font> Please choose a forum in the ",
    "<a href=\"forumlist.php$sid1\">forum list</a>.</p>\n";
  show_copyright();
  echo "</body></html>\n";
  exit;
}

If all goes well, then first the message header (author, date, etc.) is displayed as a table, followed by the message itself. Please note in particular how the PHP function nl2br transforms line breaks in the message text into HTML line breaks and how str_replace replaces sequences of blank characters by nonbreaking HTML spaces with the code &nbsp; (important for the representation of indented program code!):

// show message
$fnt1s = '<font color="#b02020">';  // font 1 start
$fnt1e = '</font>';  // font 1 end
$fnt2s = '<font color="#303030">';  // font 2 start
$fnt2e = '</font>';  // font 2 end
$row = mysql_fetch_object($result);
echo "<table>\n";
echo "<tr><td>$fnt1s Subject:$fnt1e</td><td><b>", $fnt2s,
  htmlentities($row->subject), $fnt2e, "</td></b>\n";
echo "<tr><td>$fnt1s Author:$fnt1e</td><td>", $fnt2s,
  htmlentities($row->username), $fnt2e, "</td>\n";
echo "<tr><td>$fnt1s Date:$fnt1e</td><td>",
  "$fnt2s$row->timest$fnt2e</td>\n";
echo "<tr><td>$fnt1s Forum:$fnt1e</td><td>", $fnt2s,
  htmlentities($row->forumName), $fnt2e, "</td>\n"; echo "</table>\n";
$msg = nl2br(htmlentities($row->msgText));
$msg = str_replace(" ", "&nbsp;&nbsp;", $msg);
echo "<p>$msg</p>\n";

Links to forumthread.php and forumwrite.php are displayed at the bottom of the page:

// link to forumthread.php (display entire thread)
if($row->rootID)
  $query = "rootID=$row->rootID" . $sid2;
else
  $query = "rootID=$row->msgID" . $sid2;
echo "<p>",
  build_href("./forumthread.php", $query, "Show entire thread"),
  ".\n";
// link to forumwrite.php (reply to message)
$query = "forumID=$row->forumID&msgID=$row->msgID" . $sid2;
echo " / ",
  build_href("./forumwrite.php", $query, "Reply to this message"),
  ".\n";
// link to forumwrite.php (compose new message)
$query = "forumID=$row->forumID" .$sid2;
echo " / ",
  build_href("./forumwrite.php", $query, "Write a new message"),
  ".</p>\n";
?>
</body></html>


Displaying a Thread (forumthread.php)

The script forumthread.php is to some extent a combination of forumread.php and forummessage.php: With it, all messages of a thread—first the titles and then the actual text—are displayed on a single HTML page. This makes it possible to read the entire thread without having to jump back and forth between pages.

The URL variable rootID is passed to the script. After a brief test as to whether this variable contains reasonable data, a SELECT query is employed to read all the messages of the thread:

<?php // php/myforum/forumthread.php
   . . .
$result =
  mysql_query("SELECT messages.msgID, subject, msgText, " .
             "  level, rootID, username, forumName, " .
             "  DATE_FORMAT(timest, '%Y/%c/%e %k:%i') AS timest " .
             "FROM messages, users, forums " .
             "WHERE (messages.rootID = '$rootID' " .
             " OR  messages.msgID = '$rootID') " .
             "  AND messages.userID = users.userID " .
             "  AND messages.forumID = forums.forumID " .
             "ORDER BY orderstr");
 . . .

Otherwise, the script contains no code that is not similar to code that we have already described elsewhere. Therefore, we shall not include it here.


Contributing to a Discussion (forumwrite.php)

The script forumwrite.php enables the user to compose a new message in a discussion group or to reply to an existing message (see Figure 13-7). The program code is responsible both for the display of the form and for the evaluation of the form data.

Program Structure

Variables

The script expects two URL variables as parameters:

  • forumID contains the ID number of the discussion forum.

  • msgID contains the ID number of the message that is to be replied to. If msgID is absent, the script assumes that a new message is being composed (that is, a new thread is to be created).

The form data are read from the following variables:

  • formForumID and formMsgID contain the ID numbers of the forum and the message that is to be replied to. (Both variables are represented in the form with <input type="hidden" ... >.)

  • formSubject contains the title of the message.

  • formText contains the message text.

Initialization

The code begins as usual with the initialization of the passed variables:

<?php // php/myforum/forumwrite.php
include("myforumconnect.inc.php");
// URL variables:   forumID, msgID (optional)
// form variables: formForumID, formMsgID, formText, formSubject, formSubmit
$forumID        = array_item($_REQUEST, 'forumID');
$msgID          = array_item($_REQUEST, 'msgID');
$formSubmit     = array_item($_POST,       'formSubmit');
$formForumID    = array_item($_POST,       'formForumID');
$formMsgID      = array_item($_POST,       'formMsgID');
$formText       = array_item($_POST,       'formText');
$formSubject    = array_item($_POST,       'formSubject');
// remove magic quotes \ from form variables
if(get_magic_quotes_gpc()) {
  $formText       = stripslashes($formText);
  $formSubject    = stripslashes($formSubject); }
// session management
session_start();
header("Cache-control: private"); // \ in forms
$sesUserID     = array_item($_SESSION, 'sesUserID');
$sesUserName = array_item($_SESSION, 'sesUserName');
if(SID) {  . . .    } // as in  forumlogin.php

Storing a Message (insert_new_message)

In order to store the new message in the messages table, the following parameters are passed to the function insert_new_message: the ID number of the original message (or 0), the ID number of the forum, the ID number of the user, the message header, and the actual message text.

If 0 was not passed in parentID, then the function first returns information about the original message, which it stores in the variables orderstr, level, and rootID. Of particular interest is orderstr, which with the help of the PHP function bin2hex is transformed into hexadecimal format:

// auxiliary function for storing a message
function  insert_new_message($parentID, $forumID, $userID,
    $msgSubject, $msgText) {
  if($parentID) {
    $result = mysql_query("SELECT * FROM messages WHERE msgID=$parentID");
    $row = mysql_fetch_object($result);
    $level = $row->level+1;
    $orderstr = bin2hex($row->orderstr);
    $rootID = $row->rootID;
    if(!$rootID) // if rootID=0, use msgID as rootID for new message
  $rootID = $row->msgID;
  }
else {
  $level=0; // new message: level=rootID=parentID=0
  $rootID=0;
  $parentID=0;
}

If the maximal hierarchical level within a thread has not been reached, then the character string orderstr is now extended by the hexadecimal code of the current timestamp value:

// create orderstr character string (append hex code of the current timestamp);
maximum level = 31
if($level<=31) {
  $tmp = dechex(time());
  if(strlen($tmp)<8)
      $tmp = str_repeat(" ", 8-strlen($tmp));
  $orderstr = "0x" . $orderstr . $tmp;
}
else {
    $level=31;
    $orderstr = "0x" . $orderstr; }

The data, after having been thus prepared, must now be stored:

  mysql_query("INSERT INTO messages (forumID, parentID, rootID, " .
    "userID, subject, msgText, level, orderstr) " .
    "VALUES($forumID, $parentID, $rootID, $userID, " .
    " '" .  addslashes($msgSubject) . "', " .
    " '" .  addslashes($msgText) . "', " .
    "  $level, $orderstr)");
}

Calling insert_new_message

When the script for evaluating form data is called, first a test is made as to whether formMsgID actually refers to an existing message. If that is the case, then the message is stored with the function insert_new_message, which was described above. Then a shift to page forumread.php is carried out, where the new message is to be displayed.

if($formSubmit=="OK" and $formSubject
  and $formText and $formForumID
  and $sesUserID) {
  // is formMsgID (if specified) a valid ID number?
  $invalidMsgID=0;
  if($formMsgID) {
    $result = mysql_query("SELECT msgID FROM messages" .
      "WHERE msgID = '$formMsgID' " .
      "LIMIT 2");
    if(!$result or mysql_num_rows($result)!=1)
  $invalidMsgID=1;
  }
  // store message
  if(!$invalidMsgID)
    insert_new_message($formMsgID, $formForumID, $sesUserID,
      trim(stripslashes($formSubject)),
      trim(stripslashes($formText)));
  // display new message in forumread.php
  header("Location: ./forumread.php?forumID=$formForumID$sid2");
  exit;
}
?>

Displaying the Form

The remaining code is responsible for displaying the form. First, however, some tests are necessary: Does sesUserID contain an ID number? If not, the user is taken to the login form.

Is msgID correct? If so, then the subject and text of the message for which an answer is to be written are determined. (The message text is displayed before the input form so that the author of the new message can refer to it and perhaps copy and insert parts of it. If everything is all right to this point, the query results are stored temporarily so that later, the form controls can be initialized.

Is forumID correct? If not, then an error message is displayed:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN">
<html><head>
  <meta http-equiv="Content-Type"
  content="text/html; charset=iso-8859-1" />
<title>PHP Programming, new message (myforum database)</title>
</head><body>
<h2>myforum: New message</h2>
<?php
// does sesUserID contain a value?
if(empty($sesUserID))  . . .    display error message --> forumlogin.php
// copy form variables in forumID and msgID
if($formForumID) {
  $forumID = $formForumID;
  $msgID = $formMsgID;
}
// determine original message
if($msgID) {
  $result =
  mysql_query("SELECT msgID, msgText, subject, level, rootID, " .
           "  DATE_FORMAT(timest,'%Y/%c/%e %k:%i') AS timest," .
               "  forumID, username " .
               "FROM messages, users " .
               "WHERE msgID = '$msgID' " .
               "  AND messages.userID = users.userID " .
               "LIMIT 2");
  // was msgID invalid?
  if(!$result or mysql_num_rows($result)!=1) {
               .. display error message, exit
  }
  // store query result
  $row = mysql_fetch_object($result);
  $forumID = $row->forumID;
  $oldMsg = $row->msgText;
  $oldSubject = $row->subject;
  $oldLevel = $row->level;
  $oldRoot = $row->rootID;
  $oldOrderstr = $row->orderstr;
  $oldAuthor = $row->username;
  $oldDate = $row->timest;
}
// determine name of the forum
$result = mysql_query("SELECT forumID, forumName FROM forums " .
  "WHERE forumID = '$forumID' " .
  "LIMIT 2");
// was forumID invalid?
if(!$result or mysql_num_rows($result)!=1) {
      . . .    display error message, exit
}
$forumID = mysql_result($result, 0, 0);
$forumName = mysql_result($result, 0, 1);

With this, all data are obtained for displaying the original message and then display the form:

// form header
if($msgID) {
  $fntBlue = '<font color="#0000ff">';
  echo "<p>$fntBlue", "Reply to message <i>",
      htmlentities($oldSubject), "</i></font></p>\n";
  echo "<p>$fntBlue", "Original message text:</font></p>\n";
  echo "<p>$fntBlue<i>",
    str_replace("   ", "&nbsp;&nbsp;", nl2br(htmlentities($oldMsg))),
    "</i></font></p>\n";
}
else
    echo "<p>New message for forum <i>", htmlentities($forumName),
"</i></p>\n";
// auxiliary variables for displaying the form
$actionscript = "forumwrite.php$sid1";
// message title
$newsubject = "";
if($formSubmit)
  $newsubject = htmlentities(stripslashes($formSubject));
elseif($msgID) {
  $newsubject = "Re: " .  $oldSubject;
  if(strlen($newsubject)>78)
    $newsubject = substr($newsubject, 0, 78);
}
// message text
$newtext="";
if($formSubmit)
    $newtext = $formText;
// display form
?>
<form method="post" action="<?php echo $actionscript;?>">
<table>
<tr><td>Author:</td>
        <td> <?php echo htmlentities($sesUserName); ?></td>
<tr><td>Subject:</td>
    <td> <input name="formSubject" size="60" maxlength="78"
      value="<?php echo htmlspecialchars($newsubject); ?>" /></td>
<tr><td>Message:</td>
    <td><textarea name="formText" rows="12" cols="50"><?php
      echo htmlspecialchars($newtext); ?></textarea></td>
<tr><td></td>
    <td><input type="submit" value="OK" name="formSubmit" /></td>
<input type="hidden" name="formForumID"
    value="<?php echo $forumID;?>" />
<input type="hidden" name="formMsgID" value="<?php echo $msgID;?>" />
</table></form> </body></html>


Leaving the Forum (forumlogout.php)

The logout script has the sole task of deleting the session variables. Then the login page forumlogin.php is again displayed:

session_start();
unset($_SESSION['sesUserID']);
unset($_SESSION['$sesUserName']);
header("Location: ./forumlogin.php");
exit; ?>


Possibilities for Extension

In addition to the suggestions for detailed improvements and extensions mentioned as the individual script files were presented, there are, needless to say, a number of major features that could be added or improved:

  • Administrative tasks: For the administrator of the forum there should be the simple option of generating new discussion groups or terminating existing ones, deleting individual messages, and so on. At present this is possible only via directly accessing the myforum database.

  • Moderated forum: Experience has shown that open discussion groups with a great deal of activity tend to have a small number of relevant messages littered with a large quantity of garbage. A moderated forum can rescue the situation: Every message must be approved by the moderator before it is posted. Moderation requires a password-protected site, over which the moderator can publish or delete new contributions.

  • E-mail notification of replies: An individual who posts a message to a forum is usually interested in the reactions to that posting. Instead of having to check daily (or even more often for the nervous Nellies and Nelsons among us) to see whether any responses have been posted, the poster of the original message might enjoy receiving notification by e-mail of any replies. This would enable the poster to fire off a rapid (but not, please, rabid) reply.

  • Multilingual guidance: In the forums table a language is specified for each discussion group. However, all the instructions for all of the forums are in English. Wouldn't it be nice if all of the links, form titles, and so on, appeared in the language of the discussion?

  • Search option: Currently, the forum offers no possibility of searching. It would be desirable to have a key-word search (full-text search) either for Subject lines only or for the full text of the message, as well as a search for authors (for example, display all messages posted by a particular author).

  • Personalization: In addition to the name of the user, other information could be stored, for example, a default discussion group, desired parameters (such as the number of messages to be displayed per page), and so on. It would be more complicated to implement an option whereby messages that had been read could be marked as such or marked not to be displayed again.

Tip

Let us end by repeating the little tip with which we began this chapter: There is plenty to invent besides the wheel, which has already been invented, and reinvented. If you would like to practice your PHP and MySQL, then implementing all of these features would be a good exercise. However, if your goal is to create a stable and reliable discussion forum as quickly as possible, then your time might be better spent in seeking out tested solutions:


Chapter 14: Perl

For many years, perl has been the best-beloved scripting language in the Unix/Linux universe. Moreover, Perl continues to play an important role as a programming language for CGI scripts, by which dynamic web sites can be realized. This chapter gives a brief introduction to MySQL database access with Perl and shows, by means of a few examples, the large bandwidth of possible applications.

Programming Techniques

Pointer

This chapter assumes that Perl and the modules DBI and DBD::mysql are installed on your computer and that they are functioning properly. An introduction to the installation procedure can be found in Chapter 2.

Further information on both Perl modules can be found in the official MySQL documentation (the chapter "MySQL APIs"). Furthermore, the command perldoc nets you extensive on-line help:

perldoc DBI
perldoc DBI::FAQ
perldoc DBD::mysql

HTML versions of these three help pages can be found at, for example, http://www.perldoc.com/. Links to additional Internet pages with information on database programming with Perl and DBI can be found at my home page: http://www.kofler.cc/mysql.

Finally, a compact reference to DBI functions and methods can be found in Chapter 20.

The Modules DBI and DBD::mysql

Access to MySQL is carried out in Perl via the modules DBI and DBD::mysql. (The abbreviation "DBI" stands for database interface, while "DBD" stands for database driver.) DBI is a general interface for database programming, independent of particular database systems. Thus DBI can be used for database programming with Oracle, DB/2, etc. Ideally, the code is the same in every case; that is, you can switch database systems without changing the Perl code (with the exception of datasource character strings for establishing the connection).

DBD::mysql is used by DBI to communicate with MySQL. (There exist comparable driver modules for a host of other database systems.) Which driver module DBI must use is determined by the character string specified when the connection is established (method connect). Therefore, the instruction use DBI at the beginning of the Perl script suffices for incorporating the DBI module.

Although DBI actually follows a process that is independent of the database, DBD:: mysql makes available a number of MySQL-specific functions. The reason for this is that DBI constitutes the greatest common denominator of all database systems, and thus is relatively small. The use of MySQL-specific functions simplifies MySQL programming with Perl considerably, but leads, of course, to the result that the code can be ported to another database system only with difficulty.

DBI and DBD are object-oriented modules. For this reason, most of their functions are available in the form of methods, which are applied to objects (which in Perl are represented by handles). For example, DBI->connect returns a handle to a database object (which is usually stored in the variable $dbh). Various functions that affect the database can now be executed as methods of this object, for example, $dbh->do("INSERT ... "), for executing an SQL command.

Establishing a Connection to the Database

The connection is established with the DBI method connect. The first parameter to this method is a character string specifying the type of the database and the name of the computer (or localhost). The syntax of the character string can be deduced from the following example. The next two parameters must contain the user name and password.

A fourth, optional, parameter can be used to specify numerous attributes. For example, 'RaiseError'=>1 has the effect that the Perl script is broken off with an error message if the connection to the database cannot be established. (A list of the possible attributes can be found in Chapter 20. Further information on error-handling can be found in this chapter.)

use DBI;
$datasource = "DBI:mysql:database=mylibrary;host=localhost";
$user = "root";
$passw = "xxx";
$dbh = DBI->connect($datasource, $user, $passw,
    {'RaiseError' => 1});

Specifying the Configuration File

In the datasource character string you may also specify the name of a configuration file in which the user name, password, and possible additional connection options are specified. This is particularly practical under Unix/Linux if such configuration files are stored in the home directories of the users. (Information on creating configuration files can be found in Chapter 19.)

In the following example, the option mysql_read_default_file specifies the location of the configuration file (relative to the home directory from the environment variable HOME).

The option mysql_read_default_group specifies the group [mygroup] within the configuration file. If this option is not used, then connect automatically evaluates the group [client]. The second and third parameters to the method connect are both specified as undef, to make it clear that the user name and password are specified in another location:

$datasource = "DBI:mysql:database=mylibrary;" .
    "mysql_read_default_file=$ENV{HOME}/.my.cnf;" .
    "mysql_read_default_group=mygroup";
$dbh = DBI->connect($datasource, undef, undef, {'RaiseError' => 1});

The associated configuration file ~/.my.cnf might look like the following:

[mygroup]
user=root
password=xxx
host=uranus.sol

Please note that access rights to this file are set in such a way that only the user is allowed to read it.

Remark

Under Windows, I was unable to evaluate a configuration file with DBI->connect( ). Perhaps this function is available only under Unix/Linux (though such is not explicitly documented).

Terminating the Connection

All accesses to the database are attained via the variable $dbh (or from variables derived from it). When the connection to the database is no longer needed, it should be closed with disconnect:

$dbh->disconnect();

Persistent Connections

Unlike the case of MySQL functions for PHP, in the case of the Perl DBI module there is no possibility of making use of persistent connections in order to minimize the time for repeated establishment of a connection to the database.

If you execute CGI Perl scripts via the Apache module mod_perl (which is recommended for reasons of efficiency), you achieve by use of the Perl module Apache::DBI that MySQL connections remain after the script has ended and are reused when another script requires the same type of MySQL connection. To use the module you have merely to insert use Apache::DBI before all use DBI commands in your script. No other changes are necessary. (Apache::DBI replaces the connect method of DBI with its own version.) Additional information on Apache::DBI can be found at the following location: http://www.perldoc.com/cpan/Apache/DBI.html.

Executing SQL Commands

SQL Commands Without Record List as Result

SQL commands that do not return a list of records are generally executed by means of do:

$n = $dbh->do("INSERT INTO authors (authName)". "VALUES ('New author')");

Normally, do returns the number of altered data records. Other possible return values are as follows:

  • "0E0" means that no record has been changed. This character string can be changed into a number by means of an arithmetic operation (i.e., $n+=0).

  • −1 means that the number of changed records is unknown.

  • undef means that an error has occurred.

Determining AUTO_INCREMENT Values

After INSERT commands it is frequently necessary to determine the AUTO_INCREMENT value of the newly inserted data record. For this task, the attribute mysql_insertid of DBD::mysql is helpful:

$id = $dbh->{'mysql_insertid'};
Remark

The attribute mysql_insertid is not portable; that is, it is available only for MySQL databases. In porting the code to another database system you will have to find another way of proceeding.

SELECT Queries

SQL commands that return a list of records (typically SELECT commands) cannot be executed with do. Instead, the query must first be prepared with prepare. (Many database servers will compile the query or execute other operations. However, with MySQL that is not the case.)

The return value of prepare is a statement handle, which must be used for all further operations with the query, even for the method execute, in order actually to execute the query. (If an error occurs in the execution of the query, the return value is undef.)

$sth = $dbh->prepare("SELECT * FROM titles LIMIT 5");
$sth->execute();

Then the resulting records can be output with $sth, about which we shall have more to say in the next section. When the evaluation is complete, the resources bound to $sth should be released with finish:

$sth->finish();# delete query object

SQL Queries with Wild Cards for Parameters

It often happens that the same type of query needs to be executed over and over with varying parameters (WHERE id=1, WHERE id=3, etc.). In such cases, DBI offers the possibility of replacing the parameters in prepare by a question mark, which serves as a wild card. Each question mark corresponds to a parameter, whose value must subsequently be specified with execute. Note that in execute, the correct order of the parameters must be adhered to. The following lines demonstrate how to proceed:

$sth = $dbh->prepare("SELECT * FROM titles " .  "WHERE catID=? AND publID=?");
$sth->execute(1, 1);   # title with catID=1 and publID=2
 . . .                 # evaluate results
$sth->execute(1, 2);   # title with catID=1 and publID=2
   . .                 # evaluate results
$sth->finish();        # delete query object

In working with wild cards a few points should be noted: First of all, wild cards must be given in SQL commands unquoted (even if it is a character string). Secondly, the value passed to execute by the DBI module is dealt with automatically with quote, which places character strings in single quotation marks and prefixes a backslash to the apostrophe and backslash characters. If you wish to pass NULL to an SQL command, you must specify the value undef, which quote turns into NULL.

If you execute the commands

$sth = $dbh->prepare("INSERT INTO publishers (publName)" .
                     "VALUES (?)");
$sth->execute("O'Reilly");

with Perl, then it is the following SQL command that is actually executed in MySQL:

INSERT INTO publishers (publName) VALUES ('O\'Reilly')

This example shows that prepare and execute are suitable not only for SELECT queries, but also for all SQL queries that need to be executed a number of times.

Remark

This procedure is not only elegant, but with many database servers it is also efficient. Namely, the command is temporarily stored on the server with prepare. Thereafter, only the parameters need to be transmitted to the server. However, MySQL does not implement this type of optimization.
Tip

Wild cards can also be used when SQL commands are executed with do. This does not contribute to increased speed, but it does generally improve readability, since DBI takes care of both single quotation marks and quote:

$dbh->do("INSERT INTO table (cola, colb) VALUES (?, ?),
  undef, ($data1, $data2)");

Evaluating SELECT Queries

Provided that no error occurs in a query (test by if(defined($sth))), the resulting records can be read via the statement handle (i.e., $sth). There are several different methods for accomplishing this.

Reading Data Records with fetch alias fetchrow_array

The two equivalent methods fetch and fetchrow_array return an array with the values of the next record. Within the array, the value NULL is expressed by undef. When the end of the record list is reached or if an error occurs, the array is empty. (To distinguish between these two cases, $sth->err( ) must be evaluated.) The following lines execute a query and display the results line by line. Here NULL is represented as a character string <NULL>:

$sth = $dbh->prepare("SELECT * FROM titles LIMIT 5");
$sth->execute();
while(@row = $sth->fetchrow_array()) {# process all records
  foreach $field (@row) {# each field
    if(defined($field)) {# test for NULL
  print "$field\t";
    }  else {
  print "<NULL>\t";
    }
  }
  print "\n";
}
$sth->finish();

The evaluation of the columns in a foreach loop is rather the exception. To be sure, the individual columns could also be selected in the form $row[n], where n=0 must be specified for the first column. A further variant consists in assigning all the columns at once to the variables in question. Then, of course, the correct order of the variables must be heeded:

($titleID, $title, ... )= @row;

Selecting Individual Values

It is often clear from the outset that a query will return only a single value (e.g., SELECT COUNT(*) FROM ... ). In this case, neither a loop over all records nor an evaluation of all elements of an array is necessary. Instead, simply assign fetchrow_array to a scalar variable:

$sth = $dbh->prepare("SELECT COUNT(*) FROM titles");
$sth->execute();
$result = $sth->fetchrow_array();
print "$result\n";
$sth->finish();
Caution

According to the DBI documentation, in the scalar context (e.g., $field = $sth->fetchrow_array( )), fetchrow_array should return the contents of the first column. However, tests with the available version have revealed that $field contains, to the contrary, the value of the last column. There should be no problem when there is only one column.

But in addition, the scalar evaluation of fetchrow_array is not quite unproblematic. There are now three reasons for which $field can contain the value undef: The column contains NULL in the record in question, the last record was reached, or an error has occurred.

Instead of the code given above, you could use the following shorthand version:

$result = $dbh->selectrow_array("SELECT COUNT(*) ... ");

Binding Columns to Variables

In processing a SELECT command, instead of transmitting every data record manually in variables, you can automate this step and at the same time increase the efficiency and readability of your program a bit. To do this you bind the column of the query with bind_col to individual variables. Each time that fetchrow_array is executed, the associated variables contain the value of the new record. It is necessary that bind_col be executed after execute. The return value of bind_col is false if an error occurs. Please note that column numbering begins with 1 (not, as is usual in Perl, with 0):

$sth = $dbh->prepare("SELECT titleID, title FROM titles");
$sth->execute();
$sth->bind_col(1, \$titleID);
$sth->bind_col(2, \$title);
while($sth->fetchrow_array()) {
    print "$title $titleID\n";
}
$sth->finish();
$dbh->disconnect();

Instead of binding the variables individually, you can do this for all columns at once with bind_columns. Please observe the correct sequence and number of variables:

$sth->bind_columns(\$titleID, \$title);

Determining the Number of Data Records

The DBI module provides no possibility of determining the number of records returned by a SELECT command. (This has to do with the fact that with many database servers the records are transmitted to the module only when they are needed.) If you wish to know how many resulting records there are, you have the following options:

  • You can count during the evaluation. (This variant is ruled out if you wish to know the number of records in advance.)

  • You can execute the second query with SELECT COUNT(*) ... . Depending on the application, this can be relatively costly.

  • You may use the DBI method fetchall_arrayref (see below). Thereby all resulting records are transmitted together into a local array, so that the number can easily be determined.

  • You may use the method $sth->rows( ). This method can be evaluated after execute( ). At first glance, this seems the obvious solution, but it has, in fact, a number of drawbacks:

First, the method is not portable. (According to the DBI documentation, rows may be used to determine the altered records, say after UPDATE or DELETE, but not for investigating records read with SELECT.)

Second, the method is itself available in conjunction with the DBD::mysql driver only when queries are evaluated without the attribute mysql_use_result. That is indeed the default setting, but it is not always particularly efficient.

Determining Column Names and Other Metainformation

If you wish to program a generally valid function for representing tables, you need not only the data themselves, but also metainformation about the data (column names, data types, etc.). This information is made available by DBI via various attributes of $sth:

  • $sth->{'NUM_OF_FIELDS'} returns the number of columns.

  • $sth->{'NAME'} returns a pointer to an array with the names of all columns. The same holds for $sth->{'NAME_lc'} and $sth->{'NAME_uc'}, but with the names all in lowercase or uppercase, respectively.

  • $sth->{'NULLABLE'} returns a pointer to an array whose values tell whether a column can contain NULL.

  • $sth->{'PRECISION'} and $sth->{'SCALE'} return pointers to arrays with the maximum number of characters and the number of decimal places, respectively.

  • $sth->{'TYPE'} returns a pointer to an array with numerical values that permit one to determine the data types of the columns.

The following loop demonstrates the evaluation of this information:

$sth = $dbh->prepare("SELECT * FROM testall");
$sth->execute();
for($i=0; $i < $sth->{'NUM_OF_FIELDS'}; $i++) {
  print @{$sth->{'NAME'}}[$i] . " " .
  @{$sth->{'TYPE'}}[$i] . "\n";
}
Pointer

DBD::mysql makes available some additional attributes with MySQLspecific information. These attributes are presented in their own subsection of this chapter, together with other MySQL-specific (and therefore nonportable) extensions of the DBI module.

Reading Data Records with fetchrow_arrayref

The functioning of fetchrow_arrayref is similar to that of fetchrow_array. The sole difference is that now pointers (references) to arrays instead of the arrays themselves are returned. This method returns undef if the end of the data list is reached or an error has occurred.

while(my $arrayref = $sth->fetchrow_arrayref()) {
  foreach $field (@{$arrayref}) {
  . . .    as before
}

Reading Data Records with fetchrow_hashref

The method fetchrow_hashref returns an associative array (hash) with the values of the next data record. It returns undef if the end of the record list has been reached or an error has occurred.

Access to the individual columns is effected with $row->{'columnname'}. In providing the columnname, attention must be paid to case-sensitivity. The following lines demonstrate its application:

$sth = $dbh->prepare("SELECT title, titleID FROM titles LIMIT 5");
$sth->execute();
while($row = $sth->fetchrow_hashref()) {
  print "$row->{'title'}, $row->{'titleID'}\n";
}
$sth->finish();
$dbh->disconnect();

If fetchrow_hashref is called with the optional parameter "NAME_lc" or "NAME_uc", then all the hash keys are transformed into lowercase or uppercase, respectively.

Reading All Data Records with fetchall_arrayref

A drawback to all the access methods described above is that the data records must be read sequentially, and thereafter, they are no longer available. It is thus impossible to move about to your heart's content in the list of data records. (Depending on the database system, this has at least the advantage that it is miserly with resources. Under MySQL, fetchall_arrayref does not exhibit this advantage because by default, all records that are found are immediately transmitted to the client. This can be avoided only by using the attribute mysql_use_result. More about mysql_use_result can be found further along in this section.)

If you wish to access all records in an arbitrary order and to do so multiple times, you could output the entire result of the query fetchall_arrayref. As result you obtain an array with pointers to the individual records, which themselves are arrays. Access to an individual element is then accomplished via $result->[$row][$col], where the indices begin with 0:

$sth = $dbh->prepare("SELECT titleID, title FROM titles");
$sth->execute();
$result = $sth->fetchall_arrayref();
print "$result->[2][5]\n"; # third record, sixth column
$sth->finish();
$dbh->disconnect();

The number of records and columns can be determined as follows:

$rows = @{$result}; $cols = @{$result->[0]};

An optional parameter can be passed to fetchall_arrayref, which influences both the columns to be read and the organization of the data. The following command reads only the first, fourth, and last columns:

$result = $sth->fetchall_arrayref([0,3,-1]);

The following command reads all columns, but returns a pointer to an array that contains pointers to hashes. Access to individual elements is via $result->[$row]->{'columnname'},asin, for example, $result->[3]->{'titleID'}:

$result = $sth->fetchall_arrayref({});

Our last example again returns hashes, but this time only for columns with the names titleID and title:

$result = $sth->fetchall_arrayref({titleID=>1, title=>1});

In general, it makes better sense to limit the number of columns in the SELECT command, instead of waiting until fetchall_arrayref to do so, after all the data have been extracted from the database. The method fetchall_arrayref can be executed only once per query. If you would like to execute the method more than once, you must execute execute before each call.

If the query has returned no result, then fetchall_arrayref returns a pointer to an empty array. If an error occurs during data selection, then $result contains all data read to that point. If in any event you are working without 'RaiseError' => 1, then after fetchall_arrayref,you should see to it that $sth->err( ) is evaluated.

Tip

Instead of the three methods prepare, execute, fetchall_arrayref, you can also use the shorthand form $dbh->selectall_arrayref($sql).

Character Strings, BLOBs, DATEs, SETs, ENUMs, and NULL

Altering Data

To alter data records in your database you must transmit the relevant SQL commands as character strings to do. The structure of this character string must conform to the syntax of MySQL (see Chapter 18).

The starting point for the following discussion of the various data types is the variable data, which contains the data to be stored. The contents of this variable should be placed in an INSERT command, which is stored temporarily in the variable sql.In the simplest case, it goes like this:

$sql = "INSERT INTO tablename VALUES('$data1', '$data2', ... );

  • Dates and times: To format a date or time according to the MySQL official regulations, you must use the corresponding Perl function or module (for example, gmtime( ) or TIME::Local).

  • Timestamps: Perl and MySQL timestamps have the same meaning, but different formats. A Perl timestamp (function time( )) is simply a 32-bit integer that gives the number of seconds since 1/1/1970. MySQL, on the other hand, expects a timestamp in the form yyyyddmmhhmmss or yyyy-dd-mm hh:mm:ss.

    As a rule, timestamps are used to mark the time of the most recent change. In such a case, you simply pass NULL, and MySQL automatically takes care of proper storage:

    $sql .= "NULL";

    On the other hand, if you would like to store a Perl timestamp as a MySQL timestamp, then you should use the MySQL function FROM_UNIXTIME in your INSERT or UPDATE command:

    $data = time(); // data contains the current time as a Unix timestamp
$sql .= "FROM_UNIXTIME(" . $data . ")";

  • Character strings and BLOBs: If special characters occur in a character string or BLOB, then there are frequently problems with quotation. SQL requires that the single-quote, double-quote, 0-byte, and backslash characters be prefixed by a backslash.

    If you place value on the portability of your Perl code, then you should use the method $dbh->quote( ) for quoting character strings. This method not only adds \ or \0 to the character string, but also encloses the character string in single quotation marks. Thus $dbh->quote(''ab'c'') returns 'ab\'c'. (If you execute an SQL command with wild cards, quote( ) will be used automatically.)

    $sql .= $dbh->quote($data);

    In putting together SQL commands, the Perl construct qq{}, which returns the specified character string, is often useful. Within qq{}, variables are replaced by their contents (but not quoted). The advantage of qq{} over a direct concatenation of character strings with $sql="INSERT ... " is that within qq{} the single- and double-quote characters may be used:

    $data = $dbh->quote($data);
$sql = qq{INSERT INTO table (col1, col2, col3)
        VALUES ($data, 'abc', PASSWORD("abc"))};

  • NULL: I am pleased to be able to inform you, dear reader, that $dbh->quote( ) also treats the value undef correctly, and in this case returns NULL (without a single quote):

    $sql .= $dbh->quote($data);

    If youdo not wish to use quote( ), then you might try the following:

    $sql .= defined($data) ? "'$data'" : "NULL";

    Please note that within an SQL character command you do not place NULL in quotation marks as if it were a character string.

Reading Data

The starting point for the following considerations is the variable $data, which contains a data field. For example, $data can be initialized as follows:

$sth = $dbh->prepare("SELECT * FROM titles");
$sth->execute();
@row = $sth->fetchrow_array();
$data = $row(0);

  • Timestamps: Timestamps in the form of a character string of the form 20031231235959 are contained in data. This won't get you very far in Perl. If you want to work with the timestamp value in Perl, then use the MySQL function UNIX_TIMESTAMP and formulate the corresponding SELECT query. Then for the above date data contains the value 1072911599:

    SELECT ... , UNIX_TIMESTAMP(a_timestamp) FROM ...

  • Date: For a DATETIME column, data contains a character string of the form 2003-12-31 23:59:59. For DATE columns, the time information is lacking. For the further processing of dates in Perl, it is practical in many cases here, too, to use the function UNIX_TIMESTAMP. You can obtain any format you like with the MySQL format DATE_FORMAT. The following instruction results in data containing a character string of the form 'December 31 2003'.

    SELECT ... , DATE_FORMAT(a_date, '%M %d %Y')

  • Time: For a TIME column, data contains a character string of the form 23:59:59. If you do not wish to extract hours, minutes, and seconds from this character string, then the MySQL function TIME_TO_SEC can be used to return the number of seconds since 00:00:00. Additionally, you can, of course, use any of the numerous MySQL functions for processing times. Warning: UNIX_TIMESTAMP does not work for TIME columns.

    Pointer

    In Chapter 18 you will find an overview of the MySQL functions for processing and converting dates and times.

  • NULL: Since $data is as yet undefined, it contains undef. Whether $data contains NULL can most easily be determined with the Perl function defined($data). (Do not compare $data with " " or 0 to detect NULL. Both comparisons return True, though preceded by a Perl warning.)

  • Character strings and BLOBs: Perl, in contrast to C, has no difficulties with binary data; that is, even zero-bytes within a character string are correctly handled. Thus data truly contains the exact data from the database.

    Tip

    If you wish to output character strings read from tables into HTML documents, as a rule you must use the function encode_entities from the module HTML::Entities. Thefunction escapeHTML from the module CGI can be used only if you are certain that the database contains no characters other than those of the 7-bit ASCII character set.
    Remark

    If problems arise in reading large BLOBs because the maximum amount of data per field is limited, then this limit can be set (before execute) with $dbh->{'LongReadLen'}=n.The value n specifies the maximum number of bytes.

    Warning: 0 means that long fields will not be read at all. In this case, $data contains the value undef,whichcannotbedistinguished from NULL.

Determining Elements of an ENUM or SET

The use of ENUMs and SETs presents no problems in and of itself: The values are passed in both directions as simple character strings. Note that with ENUMsno empty spaces are permitted between the comma-separated items.

When in a Perl program you wish to display the character strings of an ENUM or SET for selection (in an HTML listbox, for example), you must determine the definition of this field with the SQL command DESCRIBE tablename columnname:

USE exceptions
DESCRIBE test_enum a_enum
 Field  Type                       Null Key . . .
-----------------------------------------------------
 a_enum enum('a','b','c','d', 'e') YES      . . .

The column Type of the result table contains the required information. In the following lines of code, the character string enum('a', 'b', 'c') will first be decomposed step by step and then displayed line by line. (The algorithm assumes that the SET or ENUM character string contains no commas.)

$sth = $dbh->prepare("DESCRIBE test_enum a_set");
$sth->execute();
$row = $sth->fetchrow_hashref();
$tmp = $row->{'Type'};          # enum( . . . ) or set( . . .         )
($tmp) = $tmp =˜ m/\((.*)\)/;    # xyz('a','b','c') --> 'a','b','c'
$tmp =˜ tr/'//d;                 # 'a','b','c' --> a,b,c
@enums = split(/,/, $tmp);      # @enums[0]=a, @enums[1]=b            . .
foreach $enum (@enums) {        # output all values
    print "$enum\n";
}

DBD::mysql-specific Methods and Attributes

Whereas the DBI module represents the greatest common denominator of all database APIs, DBD::mysql contains as well various MySQL-specific functions. Their application leads to Perl code that can be transferred to other database systems only with difficulty. However, such code is often more efficient.

Pointer

This section introduces only the most important MySQL-specific methods and attributes. Chapter 20 contains a complete reference of DBD::mysql extensions.

Using rows to Determine the Number of Records Found with SELECT

The method $sth->rows( ), which was mentioned previously, is, in fact, not MySQLspecific. But only with the DBD::mysql driver does this method return, after execute, the number of records found. This holds only if queries are executed without the attribute mysql_use_result.

Determining the AUTO_INCREMENT Value

The $dbh attribute mysql_insertid has also been mentioned. It enables you to determine the AUTO_INCREMENT value of the most recently inserted data record after an INSERT command:

$id = $dbh->{'mysql_insertid'};

Determining Additional Column Information

DBI makes a number of $sth attributes available that make it possible to determine information about the columns of a SELECT result (for example, $sth-> {'NAME'}). DBD::mysql adds some additional attributes, such as 'MYSQL_IS_BLOB', to establish whether a column contains BLOBs; or 'MYSQL_TYPE_NAME',to determine the name of the data type of the column.

mysql_store_result Versus mysql_use_result

If a SELECT command is executed with prepare and execute, then usually, DBD::mysql calls the C function mysql_store_result. This means that all the data records found have been transmitted at once to the client and retained there in memory until $sth->finish( ) is executed.

If between prepare and execute you set the attribute mysql_use_result to 1, then DBD::mysql uses the C function mysql_use_result for transmitting the records. This means that records are brought from the server to the client only when they are needed. In particular, in processing large amounts of data the RAM requirement can be greatly lowered on the client side. (This advantage is lost if you use fetchall_arrayref.)

$sth = $dbh->prepare("SELECT * FROM table");
$sth->{'mysql_use_result'}=1;
$sth->execute();
 | mysql_store_result (default) | mysql_use_result
$sth->rows(); | This function returns the number of records found. | This function returns 0 or the number of records transmitted thus far.
Locking | The READ-LOCK time is minimal. | If locking is used, the table is blocked until the last record is read.
Client Memory Requirement Speed | All found records are stored at the client. Access to the first record is comparatively slow (since all records are transmitted to the client); it is very fast thereafter. | Only one record is stored at the client at one time. Access to the first record is very rapid; further ac- cess is slower than with mysql_store_result.

Error-Handling

There are several possibilities for Perl-script error-handling. The most convenient way (especially during program development) is to specify the option {'RaiseError'=>1} in establishing the connection. The result is that an error message is automatically displayed at each error. Additionally, the Perl program is terminated immediately. (This holds not only for errors during the connection, but also for all additional errors that occur in the execution of DBI methods.)

For many administrative tasks this sort of error-handling is sufficient. However, once you employ Perl for programming dynamic web pages, the termination of a script (and perhaps the display of a cryptic error message) is quite the opposite of what one might call user-friendly.

You can achieve more refined error-handling by turning off DBI's automatic response to errors and executing connect( ) with {'PrintError'=>0}. Now you must consistently evaluate the two DBI methods err( ) and errstr( ). The first of these contains the error number, or 0 if no error has occurred, while errstr( ) contains the error text corresponding to the last error to have occurred.

Remark

DBI methods return undef if an error has occurred. However, individual DBI methods sometimes also return undef in the course of their normal operation (such as when a data field contains NULL). For this reason, a simple evaluation of the return value is usually insufficient.

The methods err( ) and errstr( ) can also be applied to the DBI handles $dbh and $sth (depending on the context in which the method that caused the error was executed). Immediately after the connection has been established, err( ) or errstr( ) must be applied to DBI (since $dbh cannot be used after an unsuccessful connection attempt).

Example

The following lines show error-handling for Perl DBI code. In this example, the reaction to an error is always that the script is terminated, but of course, you can execute other instructions. Note particularly how err( ) and errstr( ) are applied to the objects DBI, $dbh, and $sth.

The code contains something particular to MySQL: With the SELECT command, error-handling occurs only after execute. With many other database systems an error can be recognized after prepare (but not with MySQL):

$datasource = "DBI:mysql:database=exceptions;host=localhost;";
$user = "root";
$passw = "xxx";
$dbh = DBI->connect($datasource, $user, $passw,
  {'PrintError' => 0});
if(DBI->err()) {
  print "error with connection: " .  DBI->errstr() . "\n";
  exit(); }
$dbh->do("INSERT INTO neu.autotest (data) VALUES (1)");
if($dbh->err()) {
  print "error with INSERT command: " .  $dbh->errstr() . "\n";
  exit(); }
$sth = $dbh->prepare("SELECT * FROM test_blob");
$sth->execute();
if($sth->err()) {
  print "error with SELECT execute: " .  $sth->errstr() . "\n";
  exit(); }
while(my $hashref = $sth->fetchrow_hashref()) {
  if($sth->err()) {
    print "error with SELECT fetch: " .  $sth->errstr() . "\n";
    exit(); }
  print length($hashref->{'a_blob'}) . "\n";
}
$sth->finish();
$dbh->disconnect();

Logging (trace)

If you suspect that the cause of an error is not in your code, but in MySQL, DBI, or DBD::mysql (or if you have no idea as to the cause of the error), it can help to display internal DBI logging information or to write it to a file. This is enabled by trace( ), which, like err( ) and errstr( ), can be applied to all DBI objects (including DBI itself).

Expected as parameters are the desired logging level (0 = none, 9 = maximal logging) and an optional file name. Without a file name, the logging data are sent to STDERR (under Windows, to the standard output STDOUT):

DBI->trace(2); # activate logging globally for the DBI module
$sth->trace(3, 'c:/dbi-trace.txt'); # logging only for $sth methods


Example: Deleting Invalid Data Records (mylibrary)

After months-long experiments with the database mylibrary, a variety of invalid records have accumulated. This is a scenario well known to most database developers, and one often forgets to clean house after the testing is complete.

In the case of the mylibrary database, the script delete-invalid-entries.pl offers help in most cases. It takes care of the following cases:

  • For book titles whose publID refers to a nonexistent publisher, publID is set to NULL.

  • For book titles whose catID refers to a nonexistent category, catID is set to NULL.

  • In the table rel_title_author all entries that refer to nonexistent book titles or authors are deleted. (This can result in titles without authors or authors without titles remaining. Such records are not deleted by the script.)

Of course, the script could be enlarged to include other tests, such as for checking the hierarchy within the categories table or for searching for invalid langID entries in the titles table.

Pointer

The SQL commands that appear in the script for searching for incorrect data are described extensively in Chapter 7.

Program Code

The script begins as usual with establishing a connection to the database. If an error occurs (then or later), the script, on account of {'RaiseError' => 1}, is simply terminated with an error message:

#!/usr/bin/perl -w
#  delete-invalid-entries.pl
use strict;
use DBI;
#  declare variables
my($datasource, $user, $passw, $dbh, $sth, $row);
#  establish connection to the database
$datasource = "DBI:mysql:database=mylibrary;host=localhost;";
$user = "root";
$passw = "xxx";
$dbh = DBI->connect($datasource, $user, $passw,
  {'RaiseError' => 1});

The SELECT query finds all titles for which publID contains an invalid value (that is, a value that does not exist in the publishers table). In the following loop, publID is set to NULL with UPDATE for all of the defective records found:

#  (1a) search for invalid publID entries in the titles table
$sth =
  $dbh->prepare("SELECT title, titleID " .
               "FROM titles LEFT JOIN publishers " .
               " ON  titles.publID=publishers.publID " .
               "WHERE ISNULL(publishers.publID) " .
               "  AND NOT(ISNULL(titles.publID))");
$sth->execute();
#  set publID of the affected records to NULL
while($row = $sth->fetchrow_hashref()) {
  print "set publID=NULL for title: $row->{'title'}\n";
  $dbh->do("UPDATE titles SET publID=NULL " .
      "WHERE titleID=$row->{'titleID'}");
}
$sth->finish();
#  (1b) search for invalid catID entries in the titles table
#  . . .    code analogous to (1a)

A little more complex is the query for searching for invalid entries in the rel_title_author table. Such entries are simply deleted in the following loop. The print commands display the affected data records on the computer monitor, where the operator ? is used to replace undef by NULL.

Remark

The following code is compatible with MySQL 3.23. Starting with MySQL 4.0, the invalid records from rel_title_author can be deleted with a single DELETE command (see Chapter 7).

#  (2) search for invalid authID or titleID entries in the
#  rel_title_author table
$sth =
  $dbh->prepare("SELECT authName, title, rel_title_author.authID, " .
               "rel_title_author.titleID " .
               "FROM rel_title_author " .
               "LEFT JOIN titles " .
               " ON  rel_title_author.titleID=titles.titleID " .
               "LEFT JOIN authors " .
               " ON  rel_title_author.authID=authors.authID " .
               "WHERE ISNULL(titles.titleID) " .
               " OR ISNULL(authors.authID)");
$sth->execute();
#  delete the affected entries in the rel_title_author table
while($row = $sth->fetchrow_hashref()) {
  print "delete rel_title_author entry:\n";
  print " title = ",
    defined($row->{'title'}) ? $row->{'title'} : "NULL", "\n";
  print " author = ",
    defined($row->{'authName'}) ? $row->{'authName'} : "NULL", "\n";
  $dbh->do("DELETE FROM rel_title_author " .
           "WHERE authID=$row->{'authID'} " .
    "  AND titleID=$row->{'titleID'}");
}
$sth->finish();
#  Program end
$dbh->disconnect();


CGI Example: Library Management (mylibrary)

Access to the database looks the same in CGI scripts as it does in Perl programs, which for administrative purposes are usually executed as stand alone. One must, however, note in the output of data from the database that special characters in character strings are coded with encode_entities according to the HTML syntax.

This section introduces two small Perl programs that access the mylibrary database:

  • mylibrary-simpleinput.pl enables the input of new book titles.

  • mylibrary-find.pl enables the search for book titles.

    Pointer

    An interesting tool for Perl CGI programmers is mysqltool. This is a collection of CGI scripts for the administration of MySQL databases (comparable with phpMyAdmin). It can be found at the following address:

    http://www.dajoba.com/projects/mysqltool/

Book Search (mylibrary-find.pl)

The script mylibrary-find.pl assists in locating books in the mylibrary database. The initial letters of the title are given in the search form. Clicking the OK button leads to an alphabetical list of all titles found, together with their authors, publisher, and year of publication (see Figure 14-1).

Click To expand
Figure 14-1: Searching for book titles

Program Structure

The script mylibrary-find.pl looks after the display of a simple form as well as the evaluation of that form (including the display of search results). The method param( ) of CGI is used to evaluate the form variable formSearch.

Program Code

The code begins with the declaration of modules and variables. The combination of strict and my helps in discovering typographical errors in variable names; DBI enables access to the database; CGI assists in the output of HTML structures; CGI::Carp displays error messages in the resulting HTML document, which is especially practical in searching for errors; HTML::Entities contains the function encode_entities, with which special characters are translated into HTML syntax:

#!/usr/bin/perl -w
#  mylibrary-find.pl
use strict;
use DBI;
use CGI qw(:standard);
use CGI::Carp qw(fatalsToBrowser);
use HTML::Entities;
#  declaration of variables
my($datasource, $user, $passw, $dbh, $search, $sql, $sth, $result,
    $rows, $i, $row, $authors);

Establishing the connection to the database offers nothing new over our previous examples in this chapter. If an error occurs, an error message (in the form of an HTML document) is displayed, and the script is terminated. If the connection succeeds, then the HTML document is opened with several methods from the CGI module:

#  establish connection to the database
$datasource = "DBI:mysql:database=mylibrary;host=localhost;";
$user = "root";
$passw = "xxx";
$dbh = DBI->connect($datasource, $user, $passw,
    {'PrintError' => 0});
#  display error message if required and terminate the script
if(DBI->err()) {
  print header(),
    start_html("Sorry, no database connection"),
    p("Sorry, no database connection"), end_html();
  exit();
}
#  open HTML Document
print header(),
    start_html("Perl programming, search form for the ",
  "mylibrary database"), "\n",
    h2("Search for titles in the mylibrary database"), "\n";

Evaluate Form, Display Search Results

If the script is called with form data, then the form variable formSearch is evaluated with param. The special characters _ and % are removed from the character string. If the variable search is not empty, then in sql an extensive SELECT command is assembled and executed. The results are transmitted with fetchall_arrayref into a two-dimensional array:

#  process form data
$search = param('formSearch');
#  remove characters _ and %
$search =˜  tr/%_//d;
if($search) {
    print p(), b("Titles beginning with ", encode_entities($search));
#  title search
$sql = "SELECT titles.titleID, titles.title, titles.year, " .
         "  publishers.publName, authors.authName " .
         "FROM titles, authors, rel_title_author " .
         "LEFT JOIN publishers " .
         " ON  titles.publID = publishers.publID " .
         "WHERE titles.titleID = rel_title_author.titleID " .
         "  AND authors.authID = rel_title_author.authID " .
         "  AND title LIKE '$search%' " .
         "ORDER BY title, titleID, authName " .
         "LIMIT 100";
$sth = $dbh->prepare($sql);
$sth->execute();
$result = $sth->fetchall_arrayref({});
$sth->finish();

The evaluation of the array begins with a test as to whether any book titles were found at all. The following loop is somewhat complicated by the fact that book titles with several authors should be displayed only once, with all of the authors. Therefore, in authors a character string of all the authors is assembled. When the last record in result is reached or if the next record refers to a titleID other than the current record, then the book title is displayed:

  # were any titles found?
  $rows = @{$result};
  if($rows==0) {
    print p(), "Sorry, no titles found."; }
  #  display titles
  else {
    #  loop over all records
    for($i=0; $i<$rows; $i++) {
  $row = $result->[$i];
  # create author list
  if($authors) {$authors .= ", ";}
  $authors .= $row->{'authName'};
  #  output title if arrived at the last record
  # in  result or if titleID for the next record refers to
  #  another value
  if($i==$rows-1 ||
    $row->{'titleID'} != $result->[$i+1]->{'titleID'}) {
    print p(),
    b(encode_entities($row->{'title'})), ": ",
    i(encode_entities($authors)), ". ",
    encode_entities($row->{'publName'}), " ",
    $row->{'year'}, ".";
    $authors = "";
  }
    }
    print p(), hr(), p(), "New search:", p();
  }
}

Display Form

#  display form
print start_form(),
    p(), "Search for title beginning with  . . .    ",
    textfield({-name => 'formSearch', -size => 20,
  -maxlength => 20}), " ",
    submit({-name => 'formSubmit', -value => 'OK'}),
    end_form();
print end_html();
#  end of program
$dbh->disconnect();

Ideas for Improvements

The code as it stands permits only a search based on the initial letters of a title. It would be desirable, of course, to allow a search by author(s), categories, and so on, ideally in the form of a full-text search. (A somewhat more refined example of book search in mylibrary was presented in Chapter 12, where PHP was used as the programming language. That program offers pagewise representation of the search results as well as cross references between results, for example to display all titles written by a particular author.)

Simple Input of New Books (mylibrary-simpleinput.pl)

This section presents a minimalist input form for new books (see Figure 14-2). In two input fields are placed the new book title and a list of authors. The script then tests which if any of the authors are already stored in the database. All as yet unknown authors are added with INSERT. Then the new title is stored as well as the requisite entries in rel_title_author for relating title and authors.

Click To expand
Figure 14-2: Simple book title input

Program Structure

The script mylibrary-simpleinput.pl displays a simple form and then evaluates it when OK is clicked. The method param( ) of the CGI module is used to transmit the two form variables: formTitle with the title, and formAuthors with the list of authors.

Program Code

As with mylibrary-find.pl, the code begins with the declaration of modules and variables:

#!/usr/bin/perl -w # mylibrary-simpleinput.pl
#!/usr/bin/perl -w
# mylibrary-simpleinput.pl
use strict;
use DBI;
use CGI qw(:standard);
use CGI::Carp qw(fatalsToBrowser);
# Declaration of variables
my($datasource, $user, $passw, $dbh, @row,
  $formTitle, $formAuthors, $titleID, $authID, $author);

The next lines of code are similar to those of the script mylibrary-find.pl, which we have just seen, and so we do not reproduce the code:

#  establish connection to the database
 . . .    as in mylibrary-find.pl
# if an  error occurs, display error message and terminate script
 . . .    as in mylibrary-find.pl
#  begin HTML document
 . . .    as in mylibrary-find.pl

Storing a Book Title and Its Authors

If the script is called with form data (these are located in param( )), there follows a quick test as to whether a title and author list have been specified. If that is not the case, then an error message is displayed. The program is continued below with the redisplay of the form:

#  evaluate form data
if(param()) {
  $formTitle = param('formTitle');
  $formAuthors = param('formAuthors');
  #  title and authors were specified
  if($formTitle eq "" || $formAuthors eq "") {
    print p(), b("Please specify title and at least one author!"); }

For storing the book title in the titles table, the INSERT command with one parameter is passed to do. As parameter, $formTitle is passed. The effect of this way of proceeding is that $formTitle (with a minimal amount of typing) is placed in single quotation marks, and all special characters are handled correctly:

#  form data are correct; store them
else {
  #  store title
  $dbh->do("INSERT INTO titles (title) VALUES (?)",
    undef, ($formTitle));
  $titleID = $dbh->{'mysql_insertid'};

The list of authors is processed in a foreach loop. For each author, a test is made as to whether the author already resides in the authors table. If that is indeed the case, then authID is read from the authors table. Otherwise, the new author is stored, and the new authID value is determined with mysql_insertid. Finally, the combination of titleID and authID must be stored in the table rel_title_author:

#  store authors
foreach $author (split(/;/, $formAuthors)) {
    # does the author already exist?
  @row = $dbh->selectrow_array("SELECT authID FROM authors " .
             "WHERE authName = " .
             $dbh->quote($author));
      # yes: determine existing authID
  if(@row) {
  $authID = $row[0]; }
  #  no: store new author and determine new authID
  else {
  $dbh->do("INSERT INTO authors (authName) VALUES (?)",
           undef, ($author));
  $authID = $dbh->{'mysql_insertid'};
  }
  #  store entry in rel_title_author table
  $dbh->do("INSERT INTO rel_title_author (titleID, authID) " .
           "VALUES ($titleID, $authID)");
}

In the HTML document a brief announcement is made that the new title was successfully stored. Then the form variables are deleted so that the next title can be input in the form:

    #  feedback
    print p(), "Your last input has been saved.";
    print br(), "You may now continue with the next title.";
    #  delete form variables (to prepare for the next input)
    param(-name=>'formTitle', -value=>'');
    param(-name=>'formAuthors', -value=>'');
  }
}

Displaying the Form

The remaining lines of code serve to display the form and close the HTML document:

print start_form(),
  p(), "Title:",
  br(), textfield({-name => 'formTitle', -size => 60,
                  -maxlength => 80}),
  p(), "Authors:",
  br(), textfield({-name => 'formAuthors', -size => 60,
                  -maxlength => 100}),
  br(), "(Last name first! If you want to specify more ",
           "than one author, use ; to separate them!)",
  p(), submit({-name => 'formSubmit', -value => 'OK'}),
  end_form();
print end_html();
#  end of program
$dbh->disconnect();

Ideas for Improvements

The script mylibrary-simpleinput.pl does not permit one to specify subtitle, publisher, category, etc. Furthermore, it is as good as completely unprotected against input errors. It is impossible to make changes to titles already stored. Thus there is enormous potential to improve this script! (A much more satisfactory input form was presented in Chapter 12. There, to be sure, the programming language PHP was used. Nonetheless, many of the techniques employed there could easily be ported to Perl.)


Chapter 15: Java, JSP, JDBC

This chapter describes the development of MySQL applications with the programming language Java and with Java Server Pages (JSP). As an interface to MySQL we will use JDBC (Java Database Connectivity) in combination with the driver Connector/J.

Introduction

Remark

All the tests for this chapter were carried out under Windows 2000 (with Sun JDK 1.4.1) and under SuSE Linux 8.1 (with Sun JDK 1.3.1, package java2-1.3.1). The connection to MySQL was made in each case with Connector/J 3.0.3. JSP was tested only under SuSE Linux, with Tomcat 4.0.4 (Java Servlet 2.3, JSP 1.2, Apache 1.3.n).

The Java Installation

To develop your own Java programs you need the Java Software Development Kit (Java SDK, or simply JDK). The most widely distributed JDK is from Sun (the creator of Java); however, there are Java implementations available from other developers, such as IBM. This chapter focuses primarily on the Sun implementation.

JDK is bundled with many Linux implementations and can be simply installed with the package manager. For Linux distributions without Java, or for working under Windows, you can download the JDK without charge from java.sun.com/downloads (about 40 megabytes). The official name is Java 2Platform Standard Edition n SDK (or J2SE n SDK) for short), where n is the current version number.

After installation you have to complete the environment variable PATH to the bin directory of the Java installation. To do this, modify the file autoexec.bat under Windows 9x/ME. Under Windows NT/2000/XP, open the dialog CONTROL

PANEL |SYSTEM (see Figure 4-1). The components of PATH are separated by semicolons. On my Windows test computer I have installed JDK in the directory c:\jdk1.4:

C:\> PATH
PATH=C:\WINNT\system32;C:\WINNT;C:\WINNT\System32\Wbem;
  C:\WINNT\Microsoft.NET\Framework\v1.0.3705;
  C:\Programs\SharpDevelop\bin;c:\mysql\bin;c:\jdk1.4\bin

With SuSE Linux 8.1, Java is located in the directory /usr/lib/SunJava2-1.3.1, where additionally, there is a link from /usr/lib/java to this directory. PATH is correctly set during the installation of the Java package and looks like this:

linux:~ $ echo $PATH
  /usr/local/bin:/usr/bin:/usr/X11R6/bin:/bin:/usr/games:
  /opt/kde3/bin:/usr/lib/java/bin:.

After installation, under Windows you can test in the command window, or under Linux in a console window, whether everything proceeded properly:

C:\>  java Ðversion
java version "1.4.1_01"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.1_01-b01)
Java HotSpot(TM) Client VM (build 1.4.1_01-b01, mixed mode)

linux:˜  #  java Ðversion
java version "1.3.1_04"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.1_04-b02)
Java HotSpot(TM) Client VM (build 1.3.1_04-b02, mixed mode)

Hello, World!

To try out the Java compiler, create the following file Hellow.java. Note that the file name must match that of the class defined in the code exactly, including case distinction:

/* example file Hellow.java */
public class  Hellow
{
    public static void  main (String[] args)
  {
    System.out.println("Hello, world!");
  }
}

To compile, change to the directory of the file Hellow.java and execute the following command:

> javac Hellow.java

As result, you obtain the file Hellow.class. This program is executed as follows:

> java Hellow
Hello World!

Connector/J Installation

Connector/J is a JDBC driver for MySQL. (JDBC stands for Java Database Connectivity.) JDBC is a collection of classes that help in the programming of database applications with Java. JDBC is independent of specific database systems. Therefore, to create a connection to a database system via JDBC, a driver specific to the database must be used. For MySQL database systems this driver is Connector/J.

Connector/J is a so-called type-4 driver, which means that it is implemented completely in Java. The version 3 of Connector/J described here assumes JDBC 2.0 (that is, at least Java 2, version 1.2). Connector/J is compatible with MySQL 3.23.n and 4.0.n.(ForMySQL 4.1, Connector/J3 is currently in development.)

Remark

Earler versions of Connector/J had the name MM.MySQL. After the programmer Mark Matthews moved over to the MySQL team, the driver was called Connector/J and is now officially supported by MySQL.

Connector/J is available under the GPL. If you wish to use Connector/J in commercial applications, the MySQL server used must be properly licensed. Alternatively, you can take out a separate client license for Connector/J (see Chapter 1).

There are other JDBC drivers for MySQL besides Connector/J, for example, for Resin (a servlet and JSP environment):

http://www.caucho.com/projects/jdbc-mysql/index.xtp
Pointer

The documentation to Connector/J is currently rather thin, consisting only of a Readme file that is installed together with Connector/J. However, you can consult any book on Java that discusses JDBC programming for the fundamentals. With Connector/J there are few MySQL-specific differences from general JDBC programming aspects. Note, however, that Connector/J does not support some methods provided by JDBC, primarily because MySQL does not yet have the corresponding functions.

Installation

Connector/J is available for download at www.mysql.com as a *.zip (Windows) or *.tar.gz file. Both Archives contain exactly the same files. (Java is platform-independent.) To install, unpack the contents of the archive into the directory of your choice. Under Windows, use the program Winzip, and under Linux the following command:

linux:~ # tar -xzf mysql-connector-java-n.tar.gz

Here is a list of the most important files and directories of the archive:

mysql-connector-java-n/

Directory with the actual library files; the rest of the entries in this list refer to this directory.

mysql-connector-java-n/ | Directory with the actual library files; the rest of the entries in this list refer to this directory.
README | The documentation to Connector/J.
/mysql-connector-java-n.jar | All Java Classes of the driver as a Java archive file; this file is the actual driver.
/com/* | All Java classes of the driver as individual files.
/org/* | The driver's Java start class under the old driver name (org.gjt.mm.mysql.Driver).

The crucial point here is that the Java runtime environment should be able to find the new library when executing programs. To ensure this state of affairs, there are several possibilities:

  • The simplest solution is generally to copy the file mysql-connector-java-n.jar into the directory java-installation-directory\jre\lib\ext, which is automatically checked during the execution of Java programs.

  • Alternatively, you can set the environment variable CLASSPATH. This variable specifies all directories in which there are classes that should be taken into account when Java programs are executed. So that Connector/J will be considered, you must add the directory in which mysql-connector-java-n.jar is located to CLASSPATH. Note that for the execution of Java programs from the current directory, CLASSPATH must also contain the path "." (that is, a period, which denotes the current directory).

    Under Windows, you can set CLASSPATH temporarily with the DOS command SET var=xxx. To set it permanently, use the dialog CONTROL PANEL |SYSTEM under Windows NT/2000/XP (see Figure 4-1), or under Windows 9x/ME, the file autoexec.bat.Components of CLASSPATH are separated by semicolons.

    Under Linux, you can set CLASSPATH with the command export var=xxx or permanently in /etc/profile. The specified directories in Classpath are separated with colons.

  • For initial testing there is also the option of copying the com and org directories from Connector/J into the local directory (that is, into the directory in which the program that you have developed resides).

You can test whether the installation of Connector/J was successful with the following miniprogram. It should run without reporting any errors:

/* example file HelloMySQL.java */
import java.sql.*;
  public class  HelloMySQL {
  public static void  main(String[] args) {
    try {
  Driver d = (Driver)
    Class.forName("com.mysql.jdbc.Driver").newInstance();
  System.out.println("OK"); }
    catch(Exception e) {
  System.out.println("Error: " +  e.toString()); }
  }
}

And If It Doesn't Work?

If something goes wrong, then the usual suspect is that the Connector/J classes have not been found. The error message usually looks something like this: java.lang.ClassNotFoundException: com.mysql.jdbc.Driver. Here are some of the possible sources of error:

  • If you have copied mysql-connector-java-3.0.n.jar into the directory jre\lib\ext, then it is possible that more than one Java interpreter resides on the computer. (It is also possible that a run-time version and a development version have been installed in parallel.) You have apparently copied mysql-connector-java-3.0.n.jar into the directory of an interpreter that is not the active one.

    Under Linux you can determine the correct directory easily with which java. Unfortunately, there is not a comparable option under Windows, since the relevant settings are located in the somewhat opaque registration database. However, you can determine with START |SETTINGS |CONTROL

    PANEL |ADD/REMOVE HARDWARE which Java versions are already installed and when installation took place.

  • If you have edited CLASSPATH, then perhaps Java is having no trouble locating Connector/J's classes, but is unable to locate the classes of your own program (error message java.lang.NoClassDefFoundError). If CLASSPATH is defined, then it must also point to the current directory.

    Under Windows, you can determine the contents of CLASSPATH in a command window with %CLASSPATH%, and under Linux in a console window with $CLASSPATH. Under Windows, the components of CLASSPATH are separated with semicolons, while under Linux it is colons that do the separation. The following commands show possible settings (where of course, you must adapt the configuration to your own installation):

    > ECHO %CLASSPATH%
.;C:\Programs\mysql-connector-java-3.0.3-beta
linux:~ # echo $CLASSPATH
.:/usr/local/mysql-connector-java-3.0.3-beta/


Programming Techniques

The following examples assume that Java SDK and Connector/J have been correctly installed. All example programs run only in text mode. Of course, Java programs with an AWT or Swing interface will look more attractive, but for demonstrating programming techniques, text mode is completely satisfactory and offers the advantage that the code remains compact and easy to read.

Pointer

This section provides only a first introduction to JDBC programming. More advanced information is to be found in books on Java that discuss JDBC.

A First Example

The following example offers a first impression of the construction of a JDBC program. The use of import java.sql facilitates access to the JDBC interfaces and classes. The Connector/J driver is loaded with Class.forName(" ... ").newInstance( ). (This must take place before a connection is established.)

The connection is established with the method DriverManager.getConnection, where the most important parameters (driver name, host name, and database name) are passed in the form of a URL; details to follow.

Before you can execute an SQL command, a Statement object must be created, whose method executeQuery returns a ResultSet object, which contains the result of the query. In the following lines, all data records are read through (method next), selectively chosen (getInt, getString, etc.), and output to the console window:

/* example SampleIntro.java */
import java.sql.*;
public class  SampleIntro
{
  public static void  main(String[] args)
  {
    try {
  Connection conn;
  Statement stmt;
  ResultSet res; // load Connector/J driver
  Class.forName("com.mysql.jdbc.Driver").newInstance();
  // establish connection to MySQL
  conn = DriverManager.getConnection(
    "jdbc:mysql://uranus/mylibrary", "username", "xxx");
  // execute SELECT query
  stmt = conn.createStatement();
  res = stmt.executeQuery(
    "SELECT publID, publName FROM publishers " +
    "ORDER BY publName");
  // process results
  while (res.next()) {
    int id = res.getInt("publID");
    String name = res.getString("publName");
    System.out.println("ID: " + id + "    Name: " +  name);
  }
  res.close();
    }
    catch(Exception e) {
  System.out.println("Error: " +  e.toString() );
    }   }   }

Upon execution, the program returns the following result:

ID: 1 Name: Addison-Wesley
ID: 2 Name: apress
ID: 9 Name: Bonnier Pocket
ID: 5 Name: Hanser
ID: 3 Name: New Riders
ID: 4 Name: O'Reilly & Associates

Establishing the Connection

Establishing the Connection with DriverManager.getConnection

As already mentioned in our introductory example, the Connector/J driver must be loaded before the connection is established. For this, execute the method Class.forName and pass the name of the driver. The method returns a Driver object, which you generally will not need. More to the point is that Connector/J is registered as a result of this call as a JDBC driver, which now can be used:

// SampleConnection1.java
// load the Connector/J driver
Class.forName("com.mysql.jdbc.Driver").newInstance();

For further programming, what is much more important is the Connection object that you create with the method DriverManager.getConnection. To this method you usually pass three character strings, containing the basic information about the database, as a URL (uniform resource locator), the user name, and the password:

// connection to MySQL
Connection conn = DriverManager.getConnection(
    "jdbc:mysql://uranus/mylibrary", "username", "password");

The construction of the URL looks like this (here split into two lines):

jdbc:mysql://[host1][,host2  . . .    ][:port]/
    [dbname][?para1=val1][&para2=val2][&para3=val3]  . . .

Tables 15-1 and 15-2 describe the components of the connection URL and the most important optional connection parameters. A complete list of parameters can be found in the Connector/J Readme file.

Table 15-1: Components of the connection URL
Component | Function
jdbc:mysql | Selects the driver.
host1 | Specifies the host name (localhost if a server on the local computer is to be accessed).
host2, host3 | Specifies additional (optional) host names, to be used only if the connection to host1 is unsuccessful.
port | Specifies the port number (by default, 3306).
dbname | Specifies the default database.
para=value | Specifies optional parameters (see Table 15-2).

All further database operations proceed from the Connection object. If you wish to close the connection at the end of the program, simply use the method close on the Connection object.

Table 15-2: Optional parameters in the connection URL (with default settings)
Parameter | Function
connectTimeout=0 | Specifies in milliseconds the wait time for a connection to be established. A setting of 0 means that no timeout is used. This setting functions only since Java 2, version 1.4.
autoreconnect=false | Specifies whether the driver should attempt on its own to reconnect if the connection is broken.
maxReconnects=3 | Specifies the maximum number of reconnects.
initialTimeout=2 | Specifies in seconds the time between attempts at reestab- lishing a broken connection. This time does not hold for the first connection.
useUnicode=true | Specifies whether Connector/J should convert the character set valid in MySQL to Unicode (Java). The setting false deactivates string conversion.
characterset=none | Specifies which character set should be used for Unicode conversion. This setting is seldom necessary, since Connec- tor/J generally recognizes the correct character set on its own. You must specify the character set explicitly if you want a dif- ferent character set from that valid under MySQL to be used, or if you are running MySQL with a custom-developed char- acter set. This setting is considered only if useUnicode=true holds.
relaxAutoCommit=false | Specifies whether the driver is to accept COMMIT and ROLLBACK commands without reporting an error even if MySQL supports no transactions for the affected tables (for example, because the tables are in MyISAM format).
useTimezone=false | Specifies whether times should be converted between the client and server time zones.
serverTimezone=null | Specifies the time zone of the server via a time zone ID.

Establishing the Connection with DataSource.getConnection

Since Java 2, version 1.4, in addition to the method DriverManager.getConnection, which we have already described, there is an alternative way to establish a database connection: the DataSource class (package javax.sql). The JDBC documentation recommends this way, because it allows for the use of advanced functions, such as connection pooling and the execution of distributed transactions (that is, database operations that process data from several databases).

The following lines do not go into these additional possibilities, but simply show the principles of establishing the connection. To do this, you generate an object of the class com.mysql.jdbc.jdbc2.optional.MysqlDataSource and with the methods setServerName and setDatabaseName, set the host and database names. The actual connection is then established with getConnection, where you have to pass only the user name and password. This method returns a Connection object, which in the DriverManager variant is the starting point for all further database operations.

If you wish to set additional connection parameters, you can use the method setURL (the format of the URL, as described previously) or pass a Properties field to getConnection:

// SampleConnection2.java
import java.sql.*;
import javax.sql.*; // for DataSource
public class  SampleConnection2
{
  public static void main(String[] args)
  {
    try {
  com.mysql.jdbc.jdbc2.optional.MysqlDataSource ds;
  Connection conn2;
  // establish connection with a DataSource object
  ds = new com.mysql.jdbc.jdbc2.optional.MysqlDataSource();
  ds.setServerName("uranus");
  ds.setDatabaseName("mylibrary");
  conn2 = ds.getConnection("root", "xxxx");
  // proceed as previously  . . .
    }
    catch(Exception e) {
  System.out.println("Error: " +  e.toString() );
    }
  }
}
Pointer

If you wish to use more advanced JDBC functions, you should look at the Connector/J driver directory com\mysql\jdbc\jdbc2\optional.Thereyou will find, for example, the Java source texts for the Connector/J classes MysqlPooledConnection and MySQLConnectionPoolDataSource.

Problems in Establishing a Connection

Normally, the connection between a client program and the MySQL server is established via TCP/IP when the programs are running on different computers. On the other hand, if both programs are running on the same computer and a Linux/Unix system is being used, then the connection is established over a socket file, which is more efficient than TCP/IP.

However, Java does not support sockets, and so the connection is always over TCP/IP. The importance here is that in MySQL access control, the name localhost is not used as host name, but instead, the actual name of the computer. This state of affairs frequently leads to difficulties, with an error message that usually looks something like this:

linux:˜/  java SampleConnection1
Error: java.sql.SQLException: Server configuration denies access to data source

What is particularly irritating is that a local connection using the same access information (e.g., with mysql -h localhost -u name -p) works just fine. The reason is that mysql deals with local access by default over socket files, but Java does not.

The following points should help you in solving such problems. The starting point for our example is a MySQL server that is running on the computer uranus.sol with IP number 192.168.0.2. The Java program is to run on the same computer.

  • It can be that for security reasons, every network connection to the MySQL server is forbidden. (For this, at server start the option --skip-networking must be passed, or else this option must be entered in my.cnf.) Whether such is the case can be determined with the following command:

    linux~/$ mysql -h uranus.sol -u name -p databasename
Enter password: xxxxx
ERROR 2003: Can't connect to MySQL server

    The only solution is to remove this option.

  • It can also be the case that network access is indeed allowed, but the table mysql.user, which is responsible for access control, contains an invalid entry in the column hostname. This case can also be recognized with mysql. (Note that this time, the error message is different.)

    linux~/$mysql -h uranus.sol -u name -p databasename
Enter password: xxxxx
ERROR 1130: Host '192.168.0.2' is not allowed to connect to this MySQL server

    With the command resolveip 192.168.0.2 you can determine which host name is expected by MySQL for this address. (On my computer, the command returns the result uranus.sol.) Then you can check whether the table mysql.user contains a corresponding entry (and not, say, simply uranus).

  • Another possible cause is that there is a correct entry in mysql.user, but the specified user name and password do not match. Note that this time, the error message looks different:

    linux~/$ mysql -h uranus.sol -u name -p databasename
Enter password: xxxxx
ERROR 1045: Access denied for user: 'root@192.168.0.2' (Using password: YES)

    Once you have succeeded in gaining access to the MySQL server with the program mysql by giving the computer name, the Java program should generally run if the same user name and password are used.

Pointer

Further tips on solving access problems (independent of Java) can be found in Chapter 9.

Executing SQL Commands

Before you can execute SQL commands, you need a Statement object. You obtain it with conn.createStatement:

Statement stmt = conn.createStatement();
Pointer

If you have in mind executing SELECT queries, then you can use two optional parameters to select the cursor type and specify whether the data resulting from SELECT are able tobechanged:

stmt = conn.createStatement(resultcursortype, resultconcurrency);

By default, the cursor type is forward only, and the data are read only. Additional details to follow.

The Statement class now offers a selection of various execute methods:

  •

executeUpdate for INSERT, UPDATE, and DELETE commands;

    •

  •

executeQuery for SELECT queries;

    •

  •

executeBatch for executing several commands that have been previously specified with addBatch as a block (which can increase efficiency);

    •

  •

execute to execute an arbitrary SQL command (where it is not known in advance what sort of command it will be).

    •

In the following pages we shall look more closely at the variants executeUpdate, executeQuery, and executeBatch.

Executing INSERT, UPDATE, and DELETE Commands

The execution of an INSERT, UPDATE, or DELETE command is quite simple: You simply pass the SQL command as a character string to the method executeUpdate:

stmt.executeUpdate(
"INSERT INTO publishers (publName) VALUES ('new publisher')");

The method executeUpdate returns an int value that specifies how many records were changed by the command.

 Pointer

If you wish to pass strings to the SQL command that contain special characters like ', ", \, andso on, you should use prepared statements, which will be discussed shortly.

Determining ID Numbers from a New Data Record (AUTO_INCREMENT)

There are several ways of determining the AUTO_INCREMENT number of newly created records. Which variant is to be preferred depends on the version of Java that youare using.

Variant 1: Starting with Java 1.4, the method getGeneratedKeys has been available. If you are working with a current version of Java, then getGeneratedKeys represents the best, most efficient, and most portable option. The method returns a ResultSet object (which will be described later when we discuss the evaluation of SELECT queries).

If there is precisely one new record, then the ResultSet object contains only one record with the sought ID number. To evaluate, execute first the method next (to address the first record of the ResultSet object) and then read the ID number with the method getInt(1) (the index 1 denotes the column of the record):

// SampleGetID1
stmt.executeUpdate("INSERT  . . .    ");
ResultSet newid = stmt.getGeneratedKeys();
newid.next();
int id = newid.getInt(1);

According to the JDBC documentation, getGeneratedKeys should also return a list of new ID numbers when INSERT commands were executed that generate several new data records. In order for JDBC to take note of new ID numbers, the optional parameter Statement.RETURN_GENERATED_KEYS must be passed to executeUpdate. However, a subsequent evaluation of ResultSets returns only a single ID number (that of the first inserted record). The reason for this lies not with Connector/J, but in the fact that in this situation, the MySQL server is unable to make the ID information available:

stmt.executeUpdate(
  "INSERT INTO publishers (publName) VALUES ('publisher1'), (' publisher2')",
  Statement.RETURN_GENERATED_KEYS);
ResultSet newids = stmt.getGeneratedKeys();
while(newids.next()) {     // returns only the ID of publisher1!
  System.out.println("ID: " +  newids.getInt(1));
}

Variant 2: If you are working with an older version of Java, you can use the method getLastInsertID of the Statement object to obtain the last AUTO_INCREMENT value. However, this method is defined specifically only for the Statement class of Connector/J, not for the Statement interface from java.sql. Therefore, you must transform the Statement object, via a cast operation, into an object of the class com.mysql.jdbc.Statement before you can use the method. (This works for PreparedStatement objects as well, which will be described further on.)

The principal disadvantage of getLastInsertID is that the resulting code is not portable, functioning only for MySQL in combination with the Connector/J driver:

// SampleGetID2.java
Statement stmt = conn.createStatement();
stmt.executeUpdate(
  "INSERT INTO publishers (publName) VALUES ('new publisher')");
long id = ((com.mysql.jdbc.Statement)stmt).getLastInsertID();

Variant 3: A third possibility (the slowest!) is to execute the additional SQL command SELECT LAST_INSERT_ID( ) after the INSERT command. However, you must see to it that this command is executed in the same MySQL connection and within a transaction. If you are working with InnoDB tables, you must set conn.setAutoCommit(false) and close the transaction with conn.commit( ):

// SampleGetID3.java
conn.setAutoCommit(false);
stmt.executeUpdate(
  "INSERT INTO publishers (publName) VALUES ('new publisher')");
ResultSet newid = stmt.executeQuery("SELECT LAST_INSERT_ID()");
if(newid.next()) {
  id = newid.getInt(1);
  System.out.println("new ID = " +  id); }
conn.commit();

Evaluating SELECT Queries

In order to be able to evaluate the results of a SELECT query, you must execute the SQL command with executeQuery. This method returns a ResultSet object:

Statement stmt = conn.createStatement();
ResultSet res = stmt.executeQuery(
  "SELECT publID, publName FROM publishers " +
  "WHERE publID < 10 ORDER BY publName");

 Warning

If you have already developed JDBC programs with database drivers other than Connector/J, you should take note of the default properties of ResultSet objects:

By default, the ResultSet object cannot be edited (read only).

By default, the ResultSet object supports free navigation. This has the consequence that all data are immediately transferred from MySQL server to client, even with large query results. This behavior differs from that of many other JDBC drivers, for which the ResultSet is forward only by default and is transferred only as required.

If you wish to have ResultSet properties that differ from the default behavior, then you must specify this in the creation of Statement objects for the query. More details to follow (see the headings Forward-only ResultSetsand Variable ResultSets).

To begin, that is, immediately after the execution of executeQuery, no record is active. To address the first record, the method res.next( ) must be executed. All further records will be run through with next( ) until this method returns the result false.

For the currently active record, the individual columns can be selected with the methods getInt, getString, etc. Optionally, this method can be passed the column number (e.g., res.getInt(1) for the first column) or the column name (e.g., res.getInt("publID"). The latter variant leads to more readable code, but is slower.

The following loop selects all found records from the previously given SELECT query and displays them in the console window:

while (res.next()) {
  int id = res.getInt("publID");
  String name = res.getString("publName");
  System.out.println("ID: " + id + "    Name: " +  name);
}

Invalid and Empty Queries

If the syntax of an SQL command is invalid, then the execution of executeQuery throws an SQLException exception.

If the query returns no result (zero records), then executeQuery returns simply an empty ResultSet object (not NULL). The only way to determine whether a SELECT query has returned results is by executing res.next( ). Once this method has returned the value false,the ResultSet object is empty.

Checking for NULL

Methods such as getInt and getFloat that return elementary Java data types are incapable of differentiating 0 and NULL. To determine whether the last selected data field contained NULL, one has the method wasNull:

int n = res.getInt(1);
if(res.wasNull)
  System.out.println("n = [NULL]");
else
  System.out.println("n = " + n);

 Tip

With character strings, the use of was Null is unnecessary, since in String variables, the state null can be stored. You can therefore execute String s = res.getString( ... ) and then test whether s==null.

Navigating in the ResultSet

In addition to the method next, there are several additional methods for selecting the currently active data record: previous activates the previous record, while first and last activate the records that their names imply. With these methods, the data record pointer is directed to the place before the first or after the last record. You can test with isFirst and isLast whether the beginning or end of the record list has been reached. The method getRow returns the number of the current record (1 for the first); absolute activates a record that is specified by its number.

Determining the Number of Found Records

Since Connector/J returns by default navigable ResultSets, it is rather simple to determine the number of records: With last, the last record is activated, and then with getRow its number is determined:

res.last(); int n = res.getRow();

If you then wish to run a loop over all records, you must set the record pointer to the beginning:

res.beforeFirst(); while(res.next()) ...

If you also want to determine the number of columns, you need a ResultSetMetaData object, which we discuss next.

Determining Metadata About the ResultSet

If you do not know in advance which data your SELECT query returns (for example, if you wish to write a program that can work with arbitrary tables), you can employ the class ResultSetMetaData to return all relevant information about the SELECT result: the number of columns (getColumnCount), their names (getColumnName), their data types (getColumnType and getColumnTypeName), and the number of decimal places (getPrecision, getScale). With methods such as isNullable, isAutoIncrement, and isSigned, you can find out additional properties of each column. All of these methods require as parameter the column number about which you want information.

You obtain a ResultSetMetaData object with the method getMetaData, which is applied to a ResultSet object. The following lines show a simple application of ResultSetMetaData:

int i, n;
ResultSetMetaData meta = res.getMetaData();
n =  meta.getColumnCount();
System.out.println("number of columns: " +  meta.getColumnCount());
for(i=1; i<=n; i++) {
  System.out.println("column " + i + ": " +
    "  name: " +  meta.getColumnName(i) +
    "  data type: " +  meta.getColumnTypeName(i));
}

Forward-Only ResultSets

By default, ResultSets are freely navigable; that is, you can change the currently active record at will using previous, next, etc. This is convenient, but it has the drawback that when you execute executeQuery, all data records that are found must be transferred from the MySQL server to the Java program. This can involve not only large data sets, but also the need for the Java program to reserve a large amount of memory. When youare processing numerous data sequentially, this is not the optimal way to proceed.

If you wish to transfer the records individually, you must define the Statement object for your SELECT query as follows. (Integer.MIN_VALUE is the smallest representable integer. Connector/J requires that exactly this value be passed.)

Statement stmt = conn.createStatement(
  java.sql.ResultSet.TYPE_FORWARD_ONLY,
  java.sql.ResultSet.CONCUR_READ_ONLY);
stmt.setFetchSize(Integer.MIN_VALUE);
ResultSet res = stmt.executeQuery("SELECT  . . .    ");

 Warning

Note that you can use only next( ) for navigation in the ResultSet (that is, navigation is forward-only). Note as well that you can execute a further SQL command for the existing connection (Connection object) only when you have completely run through the ResultSet or have closed the connection with close. (You may, of course, have a second connection open.)

Variable ResultSets

If you want to edit the data in a table, you generally execute INSERT, UPDATE, and DELETE commands. However, JDBC offers another possibility: You can insert, edit, and delete data records directly in a ResultSet object; these changes are then executed in the underlying table. However, for this to function, certain conditions must be satisfied:

  •

    The Statement object must be opened with the additional parameter ResultSet.CONCUR_UPDATABLE.

    •

  •

    The SELECT query can encompass only one table (no JOINs), may not use any GROUP functions, and must include the primary index.

    •

If these conditions are satisfied, you can delete the current record from the ResultSet and the database with deleteRow.

To edit the current record, execute the methods updateInt(n, 123), updateString(n, "new text"), etc. You must then confirm these changes with updateRow.

To insert a new record, first execute moveToInsertRow. Now specify the value to be stored with updateXxx methods and confirm the insertion operation with insertRow.

If you wish to know the ID number of the new record (AUTO_INCREMENT), execute last. This ensures that the new record is also the active record. You can then select the ID column with getInt(n).

The following program first selects all records from the publisher table of the mylibrary database. Then a new record is inserted, after which it is edited and then deleted.

 Remark

Records inserted into a ResultSet can no longer be edited. Therefore, in our example program, the ResultSet is first input afresh.

// SampleChangeResultSet.java
stmt = conn.createStatement(ResultSet.TYPE_SCROLL_SENSITIVE,
ResultSet.CONCUR_UPDATABLE);
// insert new publisher, then display all publishers
res = stmt.executeQuery(
    "SELECT publID, publName FROM publishers ORDER BY publID");
res.moveToInsertRow();
res.updateString(2, "New publisher");
res.insertRow();
res.last();
int newid = res.getInt(1);
res.beforeFirst();
while (res.next())
    System.out.println(res.getString(1) + " " +  res.getString(2));
res.close();
// edit the previously inserted new publisher
res = stmt.executeQuery(
    "SELECT publID, publName FROM publishers WHERE publID = " +  newid);
res.next();
res.updateString(2, "new with another name");
res.updateRow();
//  . . .    and delete
res.last();
res.deleteRow();
res.close();

Closing a ResultSet Object

If you no longer need a ResultSet object, you should close it explicitly with close. This allows reserved memory to be returned earlier than otherwise. In the case of incompletely run-through forward-only ResultSets, the result is the release of resources blocked by the MySQL server.

Prepared Statements

Prepared statements make it possible to formulate an SQL command with placeholders. Before the execution of the command, the placeholders must be passed. The advantage of this way of proceeding is that JDBC looks after the transformation of character strings and binary data into a MySQL-compatible syntax. This includes both the addition of quotation marks at the beginning and end of strings and the transformation of special characters such as ' into \' and \ into \\ .

With many database systems, prepared statements can offer an advantage in speed, since the SQL code must be transferred only once to the database server. However, this is not yet the case with MySQL 4.0. It is only starting with version 4.1 that MySQL is better optimized in this way (where the advantages are realized only if a suitable Connector/J version is used with MySQL 4.1).

The starting point for prepared statements is a PreparedStatment object, which is generated with the method prepareStatment. Note that character-string parameters may not be enclosed in quotation marks, which is usually the case with strings (that is, VALUES (?) and not VALUES('?')).

To pass parameters, use methods such as setString, setInt, setNull, setDate, and setBinaryStream. (A complete reference is given in the class description of PreparedStatement.)

To execute the SQL command, use, depending on the type of command, the familiar methods of the Statement class, for example, executeUpdate, executeQuery, addBatch, and executeBatch. What is new is that you can no longer pass parameters to these methods: The SQL command was already defined in prepareStatement, and the parameters are set with setXxx methods.

In the following example, two new publishers are inserted into the publisher table of the mylibrary database:

// SamplePreparedStatement.java
PreparedStatement pstmt = conn.prepareStatement(
    "INSERT INTO publishers (publName) VALUES (?)");
pstmt.setString(1, "O'Reilly");     // inserts O'Reilly
pstmt.executeUpdate();
pstmt.setString(1, "\\abc\"efg");  // inserts \abc"efg
pstmt.executeUpdate();

Unicode

Internally, all character strings in Java are stored in Unicode format, that is, using a character set that contains more than 65,000 characters. MySQL 4.0 is unfortunately not yet Unicode compatible. (Unicode support will come only with version 4.1. However, it is unclear how the Java support for this will look. Work is now proceeding on a new version of Connector/J.)

Transforming Unicode Character Strings into the Character Set of the MySQL server

By default, Connector/J transforms all Java character strings that become components of SQL commands from Unicode into the character set that the MySQL server uses (by default, latin1; see Chapter 10). Conversely, strings read from MySQL are transformed into Unicode.

 Remark

As the basis for character set transformation, Connector/J attempts to recognize the character set of the MySQL server at the time of creation of the connection. This has functioned reliably since Connector/J 3.0.3. With earlier versions or if for some reason you wish to use a different character set, at connection time you must use a URL of the following pattern. (The string is divided here into two pieces.)

jdbc:mysql://hostname/databasename?useUnicode=true&
  characterEncoding=ISO-8859-15

A list of character sets supported by Java can be found in the Java documentation, for version 1.4.1, for example, at the address below. In establishing the connection, the character set name of the second column must be specified (Canonical name for java.io and java.lang API):

http://java.sun.com/j2se/1.4.1/docs/guide/intl/encoding.doc.html

The automatic adaptation of the character set has the drawback that all characters that cannot be represented in the character set of the MySQL server are replaced by question marks.

In the following example, three records are stored in the table test_text:

// example SampleUnicode.java
Statement stmt = conn.createStatement();
stmt.executeUpdate(
    "INSERT INTO test_text (a_varchar) VALUES ('ÄÖÜ')");
stmt.executeUpdate(
    "INSERT INTO test_text (a_varchar) VALUES ('EURO \u20ac EURO')");
stmt.executeUpdate(
    "INSERT INTO test_text (a_varchar) VALUES ('Smiley \u263A')");

The table then contains three new records, whose a_varchar column contains the following strings. Note that in the MySQL's latin1 character set, the unrepresentable characters for the smiley face and the euro symbol were automatically represented by the character ?. The characters ÄÖÜ,on the other hand, were represented correctly:

'ÄÖÜ', 'EURO ? EURO', 'Smiley ?'

You obtain exactly the same result if you generate the new records with a PreparedStatment:

PreparedStatement pstmt = conn.prepareStatement(
    "INSERT INTO test_text (a_varchar) VALUES (?)");
pstmt.setString(1, "ÄÖÜ");
pstmt.executeUpdate();
pstmt.setString(1, "EURO \u20ac EURO");
pstmt.executeUpdate();
pstmt.setString(1, "Smiley \u263A");
pstmt.executeUpdate();

Storing Unicode Character Strings Unchanged

If you wish to overcome the restrictions of the MySQL server's character set and store all Unicode characters unchanged (without running the risk of of losing individual characters), then at the establishment of the connection, you can specify the Unicode character set explicitly, e.g., UTF8.

The main disadvantage of this is that the MySQL server now has no practicable way of sorting or indexing data. Even such comparisons as SELECT ... WHERE x LIKE '%xy%' are condemned to failure:

// example SampleUnicode.java
Connection conn2 = DriverManager.getConnection(
  "jdbc:mysql://uranus/exceptions?" +
  "useUnicode=true&characterEncoding=UTF8", "root", "uranus");
Statement stmt = conn2.createStatement();
stmt.executeUpdate(
  "INSERT INTO test_text (a_varchar) VALUES ('^…
stmt.executeUpdate(
  "INSERT INTO test_text (a_varchar) VALUES ('EURO \u20ac EURO')");
stmt.executeUpdate(
  "INSERT INTO test_text (a_varchar) VALUES ('Smiley \u263A')");

Transactions

JDBC supports transactions via the Connection object. Of course, transactions are possible only if your MySQL tables are in a format that supports transactions (generally InnoDB).

By default, JDBC is in AutoCommit mode; that is, every SQL command is considered a separate transaction and is executed immediately. You can determine the mode with getAutoCommit and change it with setAutoCommit. At the same time, the starting point for the first transaction is specified by setAutoCommit(false), which you can either confirm with commit or abort with rollback. These two commands simultaneously start the next transaction.

The following lines show a simple application. Several INSERT commands are executed within a try block. If an error occurs, the entire transaction is aborted:

// no AutoCommit after each command
conn.setAutoCommit(false);
// execute several commands
try {
  stmt.executeUpdate("INSERT INTO table1 . . . ");
  stmt.executeUpdate("INSERT INTO table2 . . . ");
  conn.commit();
}
catch(Exception e) {
  conn.rollback();
}

 Remark

The InnoDB documentation recommends that in client programming you generally not rely on the transaction methods of the library or API, and instead, that you execute the SQL commands SET AUTOCOMMIT= ... and BEGIN/COMMIT/ROLLBACK explicitly. The reason is that with many libraries, the transaction functions in combination with MySQL have not been adequately tested.

Batch Commands

JDBC offers the possibility of executing several SQL commands as a block. Such command blocks are called batches in the database community. The advantage over the individual execution of SQL commands is somewhat greater efficiency (depending on the application).

 Remark

Do not confuse batches with transactions. Batches are appropriate only if you do not need to read in your altered data in the confines of the batch. Thus you can execute a series of INSERT or UPDATE commands within a batch only if you do not determine any AUTO_INCREMENT values or execute any SELECT queries in between. Moreover, batches offer no mechanism for error handling. If an error occurs within a batch, then all SQL commands executed to that point remain valid, while all further commands will be rejected.

Batches are thus a simple aid in executing a large number of INSERT or UPDATE commands efficiently. The goal of a transaction, on the other hand, is to obtain greater data security. For example, transactions ensure that either all commands of a transaction are correctly executed, or none of them.

Of course, a batch can be executed within a transaction.

Batch commands are formed in two parts: First, all commands to be executed are passed with addBatch to a Statement or PreparedStatment command. Then the commands thus collected are executed with executeBatch.

The method executeBatch returns an int field, which for each SQL command specifies how many records were changed (a number greater than or equal to 0). If this number could not be determined, but the command was executed without error, then the return value is SUCCESS_NO_INFO. If an error did occur, then the return value is EXECUTE_FAILED. (Note, however, that with syntax errors, the execution of the batch command is broken off with an exception.)

The following lines show the execution of a simple batch command:

int i;
stmt = conn.createStatement();
stmt.addBatch("INSERT INTO publishers (publName) VALUES ('publ1')");
stmt.addBatch("INSERT INTO publishers (publName) VALUES ('publ2')");
stmt.addBatch("INSERT INTO publishers (publName) VALUES " +
    "('publ3'), ('publ4')");
int[] n =  stmt.executeBatch();
for(i=0; i <  n.length; i++)
  System.out.println(
    "Recordsets changed by batch command no " +  (i+1) + ": " +  n[i]);

If no errors occur during execution, the following text appears in the console window:

Recordsets changed by batch command no 1: 1
Recordsets changed by batch command no 2: 1
Recordsets changed by batch command no 3: 2

Working with Binary Data (BLOBs)

JDBC offers several ways of dealing with binary data:

  •

    When the task at hand is to read data from a file or to store data to a file, the methods ResultSet.getBinaryStream and PreparedStatement.setBinaryStream are to be recommended. In the first case, you can read the binary data from the database with the aid of an InputStream. In the second case, you pass an existing InputStream object (e.g., that of a file) to transfer its data into a parameter of an SQL command.

    •

  •

    With BLOBs that contain Unicode text you will find getCharacterStream and setCharacterStream helpful. You can use java.io.Reader objects with them to select text at the character level.

    •

  •

    If you wish to process data as a byte array, you can use the methods getBytes and setBytes.

    •

  •

    Finally, JDBC offers the interfaces Blob and Clob (character large object): With Blob objects you can conveniently read and edit data byte by byte with the methods getBytes and setBytes. Clob objects offer a character-oriented approach with methods such as getAsciiStream and getCharacterStream. The passing of Blob/Clob objects takes place with the methods getClob, setBlob, setClob, and getBlob.

    •

An Example

The following example shows the application of the methods getBinaryStream and setBinaryStream. The program loads the content of the file test.jpg, stores it in a new record in the database exceptions (table a_blob), reads it from there, and stores the data thus obtained in the new file test-copy.jpg. Then the new record in a_blob is deleted.

The SQL commands are executed with PreparedStatement objects that are created at the beginning of the program excerpt reproduced here. For transferring the file into the a_blob table, a FileInputStream object with setBinaryStream as parameter is simply passed to the first PreparedStatement object.

It is somewhat more complicated to read the data from the a_blob table and store it in a file. The BinaryStream object obtained from the ResultSet object via getBinaryStream must be read in a loop and copied with the help of a buffer into a fileOutputStream object:

// example SampleBlob.java
import java.sql.*;
import java.io.*;
   . . .
// establish the connection
Connection conn = DriverManager.getConnection(
  "jdbc:mysql://uranus/exceptions", "root", "uranus");
// create three PreparedStatement objects
PreparedStatement pstmt1, pstmt2, pstmt3;
pstmt1 = conn.prepareStatement(
  "INSERT INTO test_blob (a_blob) VALUES(?)");
pstmt2 = conn.prepareStatement(
  "SELECT a_blob FROM test_blob WHERE id=?");
pstmt3 = conn.prepareStatement(
  "DELETE FROM test_blob WHERE id=?");
// read file test.jpg and store it in a BLOB field
File readfile = new File("test.jpg");
FileInputStream fis = new FileInputStream(readfile);
pstmt1.setBinaryStream(1, fis, (int)readfile.length());
pstmt1.executeUpdate();
fis.close();
// determine the ID of the new record
long id = ((com.mysql.jdbc.Statement)pstmt1).
// create new, empty, file copy-test.jpg
File writefile = new File("copy-test.jpg");
if(writefile.exists()) {
  writefile.delete();
  writefile.createNewFile(); }
FileOutputStream fos = new FileOutputStream(writefile);
// read BLOB field from the database
pstmt2.setLong(1, id);
ResultSet res = pstmt2.executeQuery();
res.next();
InputStream is = res.getBinaryStream(1);
// store binary data in the new file
final int BSIZE = 2^15;
int n;
byte[] buffer = new byte[BSIZE];
while((n=is.read(buffer, 0, BSIZE))>0)
  fos.write(buffer, 0, n);
// close open objects
is.close();
fos.close();
res.close();
// delete the new record
pstmt3.setLong(1, id);
pstmt3.executeUpdate();

Peculiarities of Large BLOBs

BLOBs are always transferred in their entirety between MySQL and the Java program. Connector/J is incapable of piecewise transfer. For this reason, the Java virtual machine (JVM) must be so configured that it can reserve enough space to hold the entire BLOB in local memory.

Moreover, the MySQL server must allow the transfer of large data packets (variable max_allowed_packet; see Chapter 19). If necessary, you can increase the value of the variable on the client side by executing the following command:

SET max_allowed_packet=16000000


Java Server Pages (JSP)

Tomcat Installation

Tomcat is a JSP-savvy web server. This section presents some tips for the installation and use of Tomcat 4.0.4 in combination with Apache 1.3.n under SuSE Linux 8.1.

Pointer

There is not sufficient space here to describe all possible combinations and versions of Tomcat and Apache for various Linux distributions. If you are using other Tomcat, Apache, or Linux versions, you should look at the following web sites for the documentation relating to your particular distribution:

http://jakarta.apache.org/
http://java.sun.com/products/jsp/
http://java.sun.com/products/servlet/

The basic assumption is that in addition to Java SDK, the packages apache and jakarta-tomcat have been installed. So that Tomcat can be integrated into Apache (so that Apache knows how it is to deal with JSPs), the following line must be added to /etc/sysconfig/apache. (In principle, it is also possible to use Tomcat as an independent web server, that is, without Apache; however, we will not deal with that case here.)

HTTPD_SEC_MOD_TOMCAT=yes

Furthermore, it must be ensured that both Apache and Tomcat can actually be executed:

root# /etc/init.d/apache start
root# /etc/init.d/tomcat start

To make Apache and Tomcat start automatically when the computer is started, execute the following commands:

root# insserv apache
root# insserv tomcat

You can check with a web browser to see whether everything is functioning as it should by looking at the local page http://localhost:8080.You should see the Tomcat default start page.

By default, Tomcat in the SuSE configuration uses the directory /opt/jakarta/tomcat/webapps.We should look at several examples.

Hello, JSP World!

The Java servlet specification describes, since version 2.3, exactly how a directory with JSP and servlet applications should look. There is insufficient space here to discuss this concept in detail (and furthermore, this topic has nothing to do with MySQL). The following paragraphs describe only the steps necessary for carrying out initial tests with your own JSP files. For the development of real-world JSP applications, additional configuration is necessary, affecting particularly the directory WEB-INF and the file WEB-INF/web.xml.

Pointer

Information on how to assemble and configure your JSP projects can be found in any current JSP book or, for example, at the following address:

http://jakarta.apache.org/tomcat/tomcat-4.0-doc/appdev/deployment.html

The first step in creating your own examples consists in creating a subdirectory in the webapps directory for your project:

root# cd /opt/jakarta/tomcat/webapps
root# mkdir mytest
root# cd mytest

To make Tomcat accept the directory mytest as a web application, the subdirectory WEB-INF must be created. This directory can contain various configuration files for the project as well as additional files that should not be publicly accessible over the web server. But even if WEB-INF is empty, as in this example, the directory must exist! Note also the correct spelling (including the uppercase letters).

root# mkdir WEB-INF

In order for Tomcat to become aware of the new project directory, the program has to be restarted:

root# /etc/init.d/tomcat restart

Now create a first JSP file with the name test.jsp, for example, according to the following design:

<!--  test.jsp  -->
<html>
<head><title>Hello JSP</title></head>
<body>
<h2>
  <p>Hello JSP World!</p>
  <p>Date: <%= new java.util.Date() %></p>
</h2>
</body></html>

To check whether the JSP code is being executed by the web server, give the following address to the web browser. The result should look like that depicted in Figure 15-1:

http://localhost:8080/mytest/test.jsp
Click To expand
Figure 15-1: A first JSP example

Connector/J Installation

If you wish to access MySQL in JSP code, then Connector/J must be installed as with every Java program. For this there are several possibilities:

  • Install the Connector/J *.jar file in the Java interpreter directory installation_directory\jre\lib\ext. Now Connector/J is available to all Java and JSP projects.

  • Install the *.jar file in the Tomcat directory lib or common/lib. Now Connector/J is available to all JSP projects (in common/lib,toall except for Tomcat itself).

  • Install the *.jar file in the JSP project directory webapps/jspproject/WEB-INF/lib. Now Connector/J is available only to the project jspproject. This variant is particularly attractive if your project runs on an ISP that does not wish to change its Java installation with global additional libraries.

The CLASSPATH variant discussed in the last section is not a possibility here, since Tomcat ignores this variable.

JSTL Installation

As will be described later, in JSP files you have the option of using conventional Java code or tags (such as <sql:query ..>SELECT * FROM TABLE</sql:query>). If you decide on the second option, you must install JSTL (JSP Standard Tag Library). This library is not a component of Tomcat. An archive with the JSTL for download is available at the following address:

http://jakarta.apache.org/taglibs/doc/standard-doc/intro.html

After you have downloaded the current version, you can unpack it into the directory of your choice with the following command:

root# tar xzf jakarta-taglibs-standard-n.tar.gz

As result, you obtain two directories: lib/, with *.jar files, and tld/, with *.tld files. Now you must copy those *.jar and *.tld files that you wish to deploy into the directories WEB-INF/lib or WEB-INF/. (For the examples of this chapter you will need jstl.jar and standard.jar,aswell as c.tdl and sql.tdl.)

The last step requires you to add <taglib> entries to WEB-INF/web.xml, the configuration file for your JSP project. If you do not yet have a web.xml file (which is possible only with very simple JSP projects), the following lines give a minimal version of web.xml:

<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE web-app
    PUBLIC "-//Sun Microsystems, Inc.//DTD Web Application 2.3//EN"
    "http://java.sun.com/dtd/web-app_2_3.dtd">
<web-app>
  <taglib>
    <taglib-uri>http://java.sun.com/jstl/core</taglib-uri>
    <taglib-location>/WEB-INF/c.tld</taglib-location>
  </taglib>
  <taglib>
    <taglib-uri>http://java.sun.com/jstl/sql</taglib-uri>
    <taglib-location>/WEB-INF/sql.tld</taglib-location>
  </taglib>
</web-app>

Programming Techniques

JSP files are HTML documents with embedded Java code. There are two basic ways of specifying Java code:

  • You can place traditional Java code between <% and %>. this code is executed when the page is displayed. All output carried out with system.out appears in the resulting HTML page. (This variant corresponds exactly to the way that PHP and ASP work.)

    In addition to <% xxx %>, there are some further variants: <%= xxx %> for the direct output of the result of a Java expression, <%! xxx%> for the declaration of variables that maintain their value between several page calls, and <%@ xxx%> for the definition of attributes that hold for the entire page.

  • You can also specify Java code in the form of tags, which take, for example, the form <sql:query ... >SELECT * FROM TABLE</sql:query>. These tags are given directly in HTML code and make possible much more compact code. However, such a programmed JSP page has little to do with a traditional Java program. For XML fans, on the other hand, the syntax is rather intuitive.

Example with Traditional Java Code

If you specify traditional Java code between <% and %>, then all the programming techniques for JSP programming introduced in the previous section apply. The following JSP example file page1.jsp displays all the publishers in the publisher table of mylibrary. The code corresponds almost exactly to the introductory example of the previous sections. The only difference is the output, which now takes place over the variable out, predefined in the JSP environment, and contains HTML tags. Note as well the use of the method encodeHTML. This method is necessary so that any special characters <, >, & in names of publishers are translated into correct HTML code. See Figure 15-2:

Click To expand
Figure 15-2: JSP example with traditional Java code
<!--  page1.jsp  -->
<%@ page import="java.sql.*" %>
<%@ page contentType="text/html;charset=ISO-8859-1" %>
<html><head><title>JSP + MySQL</title></head>
<body>
<%!
    public String  encodeHTML(String s) {
  StringBuffer sb = new StringBuffer();
  int slen = s.length();
  for(int i = 0; i <  slen; i++) {
    char c = s.charAt(i);
    switch(c) {
    case '<': sb.append("&lt;");   break;
    case '>': sb.append("&gt;");   break;
    case '&': sb.append("&amp;"); break;
    default:   sb.append(c); }
  }
  return sb.toString();
      }
%>
<%
  //  main code
  try {
  res;
  // establish the connection
  Class.forName("com.mysql.jdbc.Driver").newInstance();
  Connection conn = DriverManager.getConnection(
  "jdbc:mysql://uranus/mylibrary", "username", "password");
  // obtain publisher list  . . .
  Statement stmt = conn.createStatement();
  ResultSet res = stmt.executeQuery(
  "SELECT publID, publName FROM publishers " +
  "ORDER BY publName");
  //  . . .    and display
  out.println("<p>mylibrary.publishers:</p><p>");
  while (res.next()) {
    int id = res.getInt("publID");
    String name = res.getString("publName");
    out.println("<br />publID = " + id + ",      name = "
    +  encodeHTML(name));
  }
  res.close();
  out.println("</p>");
    }
    catch(Exception e) {
  out.println("<p>Error: " +  e.toString() + "</p>");
    }
%>
</body></html>

Example with JSTL Tags

If you use tags from the JSTL, then the program just given can be formulated much more simply. (The resulting web page looks exactly like that in Figure 15-2.)

To make the program understandable, we give a brief description of the tags used: <sql:setDataSource> creates the connection to the databse. As with traditional Java code, the connection information is stored as a Connection object in the variable conn.

The tag <sql:query> executes an SQL query and stores the result in the variable res. In the loop <c:forEach>, the loop variable row runs through all records of the result of the SQL query (res.rows). Access to the individual columns takes place by their names (e.g., row.publID). The output is accomplished via <c:out>. Note that the function encodeHTML used earlier is no longer required, since <c:out> takes care of the HTML-compatible coding on its own:

<!--  page2.jsp  -->
<%@ page contentType="text/html;charset=ISO-8859-1" %>
<%@ taglib uri="http://java.sun.com/jstl/core" prefix="c" %>
<%@ taglib uri="http://java.sun.com/jstl/sql"   prefix="sql" %>
<html><head><title>JSP + MySQL</title></head>
<body>
<sql:setDataSource var="conn" driver="com.mysql.jdbc.Driver"
  url="jdbc:mysql://uranus/mylibrary" user="username"
  password="password" />
<sql:query var="res" dataSource="${conn}">
  SELECT publID, publName
  FROM publishers
  ORDER BY publName
</sql:query>
<p>mylibrary.publishers:</p>
<p>
<c:forEach var="row" items="${res.rows}">
  <br />publID = <c:out value="${row.publID}" />,
  publName = <c:out value="${row.publName}" />
</c:forEach>
</body></html>

For the example to function, the Tomcat directory webapps/mytest must contain the following files:

uranus:˜ # cd /opt/jakarta/tomcat/webapps/mytest/
uranus:/opt/jakarta/tomcat/webapps/mytest # ls
.:
drwxr-xr-x        3 root    root    4096 Jan 8 16:58 WEB-INF
-rw-r--r--        1 root    root    1510 Jan 8 11:02 page1.jsp
-rw-r--r--        1 root    root     711 Jan 8 18:32 page2.jsp
-rw-r--r--        1 root    root     147 Jan 7 15:12 test.jsp

./WEB-INF:
-rw-r--r--        1 root    root   11310 Jan 8 16:54 c.tld
drwxr-xr-x        2 root    root    4096 Jan 8 16:54 lib
-rw-r--r--        1 root    root    6127 Jan 8 16:54 sql.tld
-rw-r--r--        1 root    root     561 Jan 8 17:14 web.xml

./WEB-INF/lib:
-rw-r--r--        1 root    root    20962 Jan 8 16:54 jstl.jar
-rw-r--r-- 1 root root 516487 Jan 8 16:54 standard.jar
Pointer

A briefdescription of JSTL <sql> tags can be found in the syntax reference of Chapter 20.


Chapter 16: C and C++

This chapter gives a first introduction to programming MySQL applications with the programming languages C and C++. In particular, we will introduce the client libraries for C and C++, that is, libmysqlclient and Connector/C++ (formerly known as MySQL++). This chapter considers programming only under Unix/Linux, and not under Windows.

The C Application Programming Interface

The C application programming interface (API) is the most elementary interface to MySQL. All other APIs, such as those for PHP, Perl, and C++, are based on the C API. Thus a knowledge of the C API will facilitate learning about the other APIs. The API functions constitute a component of the library libmysqlclient.Every C program that wishes to use MySQL functions must therefore have access to this library.

Please note that there are several versions of libmysqlclient in use: version 10 for MySQL 3.23.n, version 11 for the alpha and beta versions of MySQL 4.0.n, version 12 for the stable versions of MySQL 4.0.n, and finally, version 14 for MySQL 4.1.n. These libraries are largely compatible. Version 12 differs from version 10 only by some extensions that allow the new features of MySQL 4.0 to be used. This chapter is based on version 12, but because of the narrow range of difference, most of the information is valid for version 10 as well.

Assumptions

For developing C programs under Linux or a comparable Unix system, you will need, of course, the usual equipment, in particular, a C compiler (generally gcc) and the program make. Furthermore, all the necessary include and library files for compiling and linking must be installed. With many distributions (such as SuSE), certain packages must be installed as well, ending with -devel (e.g., glicb-devel). It might be a good idea to test out your installation by translating a very simple C program.

For the development of MySQL programs you will need, in addition, the MySQL developer files, consisting of include files and libraries. These files usually are located in the package MySQL-Devel-n. This chapter assumes that the developer files are located in the following places:

Include files /usr/include/mysql

Libraries /usr/lib/mysql

If these files are located elsewhere on your system, then the compile and link options will have to be changed accordingly.

Hello World

Perhaps a little program will help develop some intuition in the direction of MySQL programming with C. The following example assumes that the database mylibrary has been installed on the local computer (see Chapter 5). Furthermore, in the code you must change, in the function call mysql_real_connect, the user name ("root") and password ("XXX"). The code is documented to the extent that it should give at least some idea of what is going on. Details on the use of MySQL functions appear later, in the section on programming techniques.

Program Code

// example file hellow/main.c
#include <stdio.h>
#include <mysql.h> // functions from libmysqlclient
int  main(int argc, char *argv[])
{
    int i;
    MYSQL *conn; // connection
    MYSQL_RES *result;   // result of the SELECT query
    MYSQL_ROW row;  // a record form the SELECT query
    // create connection to MySQL
    conn = mysql_init(NULL);
    if(mysql_real_connect(
    conn, "localhost", "root", "XXX",
    "mylibrary", 0, NULL, 0) == NULL) {
  fprintf(stderr, "sorry, no database connection  . . .    \n");
  return 1;
        }
    // create list of all publishers and determine number
    // of published titles
    const char *sql="SELECT COUNT(titleID), publName \
                     FROM publishers, titles \
                     WHERE publishers.publID = titles.publID   \
                     GROUP BY publishers.publID \
                     ORDER BY publName";
    if(mysql_query(conn, sql)) {
        fprintf(stderr, "%s\n", mysql_error(conn));
        fprintf(stderr, "%s\n", sql);
        return 1;
    }
    // process result
    result = mysql_store_result(conn);
    if(result==NULL) {
        if(mysql_error(conn))
  fprintf(stderr, "%s\n", mysql_error(conn));
        else
  fprintf(stderr, "%s\n", "unknown error\n");
        return 1;
    }
    printf("%i records found\n", (int)mysql_num_rows(result));
    // loop over all data records
    while((row = mysql_fetch_row(result)) != NULL) {
        for(i=0; i <  mysql_num_fields(result); i++) {
  if(row[i] == NULL)
    printf("[NULL]\t");
  else
    printf("%s\t", row[i]);
        }
        printf("\n");
    }
    // release memory, break connection
    mysql_free_result(result);
    mysql_close(conn);
    return 0;
}

If you execute the program, you should get the following result:

uranus:˜/hellow-c $  ./hellow
 6      records found
11      Addison-Wesley
 2      apress
 1      Bonnier Pocket
 1      Hanser
 2      New Riders
 2      O'Reilly & Associates

Compiling and Linking C Programs

To compile and link the example program just presented under Linux or comparable Unix system, execute the following command:

$ gcc -o hellow -I/usr/include/mysql -lmysqlclient main.c

The following table explains the components of the command:

gcc | The GNU C and C++ compiler
-o hellow | the name of the executable code
-I/usr/include/mysql | the location of the MySQL include files (in particular, mysql.h)
-lmysqlclient | links the executable with the MySQL client library
main.c | the code file

Depending on the system configuration, the location of the MySQL include files can be different from those presented here (e.g., /usr/local/include/mysql).

Makefile

Assembling the input for the command for compiling and linking a program is generally rather labor-intensive (especially when a project consists of several files). Therefore, it is a good idea to create a file with the name Makefile that contains all necessary instructions in the syntax of the make command. To compile, you simply input make, which decides on its own which parts of the program need to be recompiled.

As starting point for creating your own projects, you can use the following example. Note that the make syntax for the indendation requires tabs (not spaces):

#  example file hellow/Makefile
CC = gcc
INCLUDES = -I/usr/include/mysql
LIBS = -lmysqlclient
all:hellow
main.o: main.c
    $(CC) -c $(INCLUDES) main.c
hellow: main.o
    $(CC) -o hellow main.o $(LIBS)
clean:
    rm -f hellow main.o

Program Development with KDevelop

If youwish to create MySQL C programs using the development environent KDevelop, you must make the following settings in the dialog PROJECT |OPTIONS:

  • COMPILER OPTIONS |FLAGS AND WARNINGS, Textbox C COMPILER FLAGS: -I/usr/include/mysql

  • COMPILER OPTIONS |LINKER FLAGS, Textbox ADDITIONAL FLAGS: -lmysqlclient -lz

Static Binding of MySQL Functions

With the Makefile presented here, the library libmysqlclient is linked to the program dynamically if the shared library libmysqlclient.so can be found in one of the usual places (under Linux in the directory /lib or /usr/lib).

If this fails (because the library was not found) or if you wish that all MySQL API functions should be integrated directly into the executable (static instead of dynamic binding), you must specify the location of the static library variant libmysqlclient.a. This is generally in the directory /usr/lib/mysql.

Furthermore, you must now specify the additional option -lz. This permits the library libz to be consulted during linking. It contains functions accessed by libmyssqlclient for compressing data exchange between client and server (when that is desired):

$ gcc main.c -I/usr/include/mysql -L/usr/lib/mysql -lmysqlclient -lz -o hellow

If youare using make,you need to change only the LIBS variable:

LIBS = -L/usr/lib/mysql -lmysqlclient -lz

It can happen (depending on the Unix system and on which libmysqlclient functions you use) that in addition to libz, you will require other libraries. The following table summarizes the options that might be necessary:

-L/usr/lib/mysql | the location of the static MySQL libraries (libmysqlclient.a)
-lc | base library with basic functions (libc)
-lcrypt | library with cryptographic functions (libcrypt)
-lm | library with mathematical functions (libm)
-lnsl | library with Name Service functions (libnsl)
-lnss_files and | library with Name Service Switch functions; further
-lnss_dns | information in man nsswitch.conf
-lz | library with compression functions (libz)

The static variant has the advantage that all libmysqlclient functions that the program uses are compiled directly into the executable. This allows the program to runona computer that does not have the libmysqlclient library installed.

However, this advantage is paid for by a larger executable than that in the dynamic version. The difference in size depends on how many functions you use from libmysqlclient. In our "Hello World" program, the dynamic executable is only 13 kilobytes, while the static variant weighs in at 175 kilobytes. (Note that only the MySQL functions are statically bound, not all functions from all other libraries.)

You can easily determine which dynamic libraries a program must access during execution with the command ldd. In the static variant of the "Hello World" program, the result looked as follows on my test system:

$  ldd hellow
      libz.so.1 => /lib/libz.so.1 (0x40020000)
      libc.so.6 => /lib/libc.so.6 (0x4002f000)
      /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)

In the dynamic variant, the list of libraries is much longer. It contains, in addition to libmysqlclient, all libraries that are accessed by libmysqlclient:

$  ldd hellow
    libmysqlclient.so.11 => /usr/lib/libmysqlclient.so.11
         (0x40020000)
    libc.so.6 => /lib/libc.so.6 (0x40057000)
    libz.so.1 => /lib/libz.so.1 (0x40175000)
    libcrypt.so.1 => /lib/libcrypt.so.1 (0x40184000)
    libnsl.so.1 => /lib/libnsl.so.1 (0x401b5000)
    libm.so.6 => /lib/libm.so.6 (0x401ca000)
    libnss_files.so.2 => /lib/libnss_files.so.2 (0x401ee000)
    libnss_dns.so.2 => /lib/libnss_dns.so.2 (0x401f8000)
    libresolv.so.2 => /lib/libresolv.so.2 (0x401fc000)
    /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)

Programming Techniques

An extensive syntax reference for the data structures and functions from the library libmysqlclient mentioned in this section can be found in Chapter 20.

Establishing the Connection

Before establishing a connection, you must initialize a data structure of type MYSQL with mysql_init. The connection is then established with mysql_real_connect, where as parameter, a pointer to the MYSQL structure as well as the usual connection data must be specified: the host name (or "localhost") or the IP number of the server, the user name, password, name of the default database, the socket number (or 0 for the default value), the name of the socket file (or NULL for the default socket file), as well as certain flags that describe particular properties of the connection. (A list of all possible constants can be found in Chapter 20.)

MYSQL *conn;
conn = mysql_init(NULL);
if(mysql_real_connect(conn, "hostname", "username", "password",
      "databasename", 0, NULL, 0) == NULL) {
        fprintf(stderr, "error message  . . .    \n");
}

Once the connection is no longer needed, it should be closed with mysql_close:

mysql_close(conn);

Evaluation of the Configuration File my.cnf

If in establishing the connection you wish to have the settings in the configuration file my.cnf automatically evaluated (e.g., connecting over a nonstandard port number or socket file), then add a call to the function mysql_options between mysql_init and mysql_real_connect. The following instruction yields the result that all configuration settings of the group [client] are considered, to the extent that NULL or 0 is passed for the corresponding parameters in the following mysql_real_connect call. (This holds for user name and password as well.)

conn = mysql_init(NULL);
mysql_options(conn, MYSQL_READ_DEFAULT_GROUP, "");
mysql_real_connect(  . . .    );

If in addition to the [client] options, you also want a further options group to be considered, then you should give its name in the last parameter of mysql_options:

mysql_options(conn, MYSQL_READ_DEFAULT_GROUP, "mygroup");

Evaluation of Options in the Command Line

With many standard MySQL programs (such as mysql, mysqladmin, mysqldump), you can send a connection parameter in a uniform format in the command line, for example, -hname or --host==name. An evaulation of this command parameter is especially sensible if you are developing administration programs in C that are to be applied to arbitrary databases. Unfortunately, the MySQL library does not offer a convenient means of doing so like mysql_options. Instead, it must be done by hand. The following points sketch the way to proceed:

  • Instead of mysql_options, you call the unfortunately undocumented MySQL function load_defaults to read the settings from my.cnf and to copy them into the field of the command parameters (argc and argv of main). This has the advantage that all settings, whether they come from my.cnf or were passed as parameters, can be processed uniformly. Also, the string field groups specifies which options groups from my.cnf should be read and copied into argv:

    const char *groups[] = {"mygroup", "client", NULL};
load_defaults("my", groups, &argc, &argv);

  • Nowcreate a loop and evaluate, with the standard function getopt_long (include file getopt.h), all parameters from argv and initialize variables for the host name, user name, etc. If a password is to be passed in the command line, you should overwrite it with blank characters for reasons of security. For interactive input of a password you can use the function get_tty_password.

  • Finally, pass the values thus determined to mysql_real_connect.

    Pointer

    This way of proceeding is described in full in the chapter "The MySQL C API" in the book MySQL by Paul DuBois. This chaptercan also be foundasa sample chapter on the author's web site, most recently at the following address:

    http://www.kitebird.com/mysql-book/

Executing SQL Commands

To execute an SQL command, pass it as a character string to the function mysql_query. The string must contain only one SQL command, and it may not end with a semicolon. The function returns 0 if the SQL command was accepted by the server. (The return value allows one to draw conclusions only about whether the command was syntactically correct, not about whether the command changed data or whether, say, a SELECT command returned results.)

After DELETE, INSERT, or UPDATE commands, you can use mysql_affected_rows to determine how many records were changed as a result of the command. The result has the data type my_ulonglong. This data type is defined in the MySQL library; it describes a 64-bit unsigned integer.

If you have inserted a new record—perhaps with INSERT—into a table with an AUTO_INCREMENT column, you can determine the value of the AUTO_INCREMENT field with the function mysql_insert_id. This function also returns a result in the form of a my_ulonglong integer:

mysql_query(conn, "INSERT INTO publishers (publName) VALUES ('publisher')");
printf("publID = %i\n", (int)mysql_insert_id(conn));

Evaluating SELECT Results

After a SELECT command executed with mysql_query, you can process the expected results with either mysql_store_result or mysql_use_result. Both functions return a MYSQL_RES structure, from which you will be able to read out all the records with mysql_fetch_row one after another. The following lines show the general way of proceeding:

MYSQL_RES *result;   // result of a  SELECT    query
MYSQL_ROW row;  // a record of the result
. . .
result = mysql_store_result(conn);
if(result==NULL)
    fprintf(stderr, "%fehler  . . .    \n");
else {
    while((row = mysql_fetch_row(result)) != NULL) {
      // process each record
    }
    mysql_free_result(result);  // release result
}

The MYSQL_RES structure for the results should then be released as soon as possible with mysql_free_result in order to keep memory requirements in the client (mysql_store_result) and resource demands on the server (mysql_use_result) as small as possible.

As the following table shows, mysql_store_result and mysql_use_result differ fundamentally internally:

mysql_store_result | mysql_use_result
All found records are immediately transmitted to client memory. With large record lists, this can result in large memory requirements for the client. | At first, no records are transmitted; that is, the data remain on the server. This means that the server is responsible for record management. Until this process is complete, resources on the server are occupied.
The records can be addressed in any order and as often as desired (function mysql_data_seek). | The records must be processed one at a time, and can be read only once.
The number of records found by SELECT can be determined immediately with mysql_num_rows. | The number of found records is known only after all records have been run through with mysql_fetch_row.

If you expect smaller rather than larger SELECT results, then mysql_store_result is the better choice. It makes less of a demand on the server and offers more convenience for client programming. Only when you expect large quantities of data (in the megabyte quantity) should you consider mysql_use_result.

Once you have read a record with mysql_fetch_row into a MYSQL_ROW structure, you can access the individual fields (columns) easily with row[n].All results are passed as 0-terminated strings (including numbers, dates, TIMESTAMPs, etc.). For fields that are empty in the database (NULL), row[n] also contains NULL. One must be careful in the evaluation of binary data, as we shall discuss a bit later.

The following loop runs through all records and outputs the individual columns, separated by tab characters, in the console window:

int i;
while((row = mysql_fetch_row(result)) != NULL) {
  for(i=0; i <  mysql_num_fields(result); i++) {
    if(row[i] == NULL)
  printf("[NULL]\t");
    else
  printf("%s\t", row[i]);
  }
  printf("\n");
}

If you are developing a program to read arbitrary tables, that is, if you do not know in advance how many columns the result will include and what data types were originally in the MySQL database, you can use mysql_num_fields to determine the number of columns, and mysql_fetch_fields to get metainformation about each column (data type, maximum number of characters and digits, etc.). These functions are described in greater detail in Chapter 20.

Error-Handling

Most MySQL functions return an error number, or 0 if no error has occurred. With some functions that are supposed to return data structures (e.g., mysql_store_result, mysql_fetch_row), the situation is reversed: Here the return value NULL means that there were problems during execution.

If you have discovered an error in this way, you can use the functions mysql_errno and mysql_error to determine the associated MySQL error number or error message as a character string. The functions return 0 or an empty string ("" )ifno error occurred:

if(mysql_query(conn, sql)) {
  fprintf(stderr, "%s\n", mysql_error(conn));
  fprintf(stderr, "error number %i\n", mysql_errno(conn));
   . . .
}

Working with Binary Data and Special Characters

Particular care needs to be taken in executing SQL commnds that contain zero bytes or special characters (such as the processing of BLOBs) or in situations in which you expect such data as the result of a SELECT query.

The function mysql_real_escape_string serves to mark problematic characters with a backslash (\0, \b, \t, \", \', etc.). This is accomplished by copying the starting character string into a new string, for which sufficient memory must be reserved in advance:

  // char *s1  target string
  // char *s2  start string (may contain zero bytes)
// unsigned long len2   the length of s2
mysql_real_escape_string(conn, s1, s2, len2);

The function mysql_real_escape_string should be used in assembling SQL commands when individual strings can contain special characters. The following lines give an example of how an INSERT command can be correctly assembled. (The resulting command looks like this: INTO publishers(publName) VALUES ('O\'Reilly').)

The example also shows the use of the auxiliary function strmov, which is also available from the MySQL library. It does the same thing as strcpy, but it returns a pointer that points to the end of the copied character string. Thus strmov is particularly well suited for assembling strings piece by piece (which under C is a particularly tiresome chore). To use strmov,itis necessary that include instructions for my_global.h and m_string.h be added (and in fact, in the order given below).

If you alter the code given below, you should ensure that the string buffer tmp is large enough for your requirements. Insert the requisite checks:

#include <my_global.h>  // necessary so that strmov works
#include <m_string.h>  // necessary so that strmov works
#include <mysql.h>
. . .
char tmp[1000], *tmppos;
char *publname = "O'Reilly";
tmppos = strmov(tmp, "INSERT INTO publishers (publName) VALUES ('");
tmppos += mysql_real_escape_string(
    conn, tmppos, publname, strlen(publname));
tmppos = strmov(tmppos, "')");
*tmppos++ = (char)0;
mysql_query(conn, tmp);

To execute SQL commands that contain zero bytes, instead of mysql_query, you must use the function mysql_real_query. The difference is that the length of the SQL string is no longer determined by the first zero byte, but is given explicitly:

// sql points to the string with the SQL command
// n specifies the length of the string
mysql_real_query(conn, sql, n);

The reading of binary data also poses problems, since the string functions usually used assume that the data in row[i] end with the first zero byte. If it is not certain that such is the case with your data, then you must determine the actual size of the data with a call to the function mysql_fetch_lengths. This function returns an unsigned long field whose values specify the size for every column of the current record. The following example shows how the function is used.

Storing Binary Data in a Database and Reading It

With the following lines, the content of the file test.jpg from the current directory is stored in the table test_blob of the database exception (see Chapter 5). Then this record is read from the table and the result stored in a new file, test-copy.jpg. The new record can now be deleted. (Note that for the sake of simplicity, the program uses a fixed buffer of 512 kilobytes for transferring data to the database. There is no check whether test.jpg is smaller than this buffer. In a real application, you should, of course, adapt the memory size dynamically.)

// example file blob/main.c
#include <stdio.h>
#include <my_global.h>   // for strmov
#include <m_string.h>     // for strmov
#include <mysql.h>
int  main(int argc, char *argv[])
{
  int id;
  FILE *f;
      MYSQL *conn;  // connectionto MySQL
  MYSQL_RES *result;   // SELECT result
      MYSQL_ROW row;  // a record of the SELECT result
  size_t fsize;
  char fbuffer[512 * 1024];   // file size maximum 512 kilobytes
  char tmp[1024 * 1024], *tmppos;
  unsigned long *lengths;
  // establish the connection
  conn = mysql_init(NULL);
  if(mysql_real_connect(
      conn, "localhost", "root", "uranus",
      "exceptions", 0, NULL, 0) == NULL) {
  fprintf(stderr, "sorry, no database connection  . . .    \n");
  return 1;
      }
  // read file test.jpg store in exceptions.test_blob
  f =  fopen("test.jpg", "r");
  fsize = fread(fbuffer, 1, sizeof(fbuffer), f);
  fclose(f);
  tmppos = strmov(tmp, "INSERT INTO test_blob (a_blob) VALUES ('");
  tmppos += mysql_real_escape_string(
      conn, tmppos, fbuffer, fsize);
  tmppos = strmov(tmppos, "')");
  *tmppos++ = (char)0;
  mysql_query(conn, tmp);
  id = (int)mysql_insert_id(conn);
  // read the new record and store the new record
  // in the new file test-copy.jpg
  f =  fopen("test-copy.jpg", "w");
  sprintf(tmp, "SELECT a_blob FROM test_blob WHERE id = %i", id);
  mysql_query(conn, tmp);
  result = mysql_store_result(conn);
  row = mysql_fetch_row(result);
  lengths = mysql_fetch_lengths(result);
  fwrite(row[0], 1, lengths[0], f);
  fclose(f);
  // delete the new record
  sprintf(tmp, "DELETE FROM test_blob WHERE id = %i", id);
  mysql_query(conn, tmp);
  // free resources
  mysql_free_result(result);
  mysql_close(conn);
  return 0;
}


Connector/C++ (MySQL++)

Forfriends of object-oriented programming, there is also an application programming interface (API) for C++ with Connector/C++ (formerly MySQL++). However, those who expect that this will make programming easier are likely to be disappointed. This is due primarily to the size of the API, which currently (version 1.7.9) consists of over sixty structures and classes with innumerable public methods. The result is that for every operation there are at least a dozen ways of performing it. (I admit that I am exaggerating a bit.) As a result, this section can offer only a couple of examples that constitute considerably less than a systematic description. For that, I would have had to write a hundred-page chapter, and for that, there is alas no room in this book.

The second great shortcoming in Connector/C++ is that although the documentation on the MySQL server www.mysql.com is extensive and conveniently example-oriented, there is no proper reference on classes, their methods, and, above all, the logic of their organization. (There is, in fact, a reference, but it seems to have been generated some time ago automatically from the header files; the reference is not complete, and it fails to provide a good understanding of the classes and their methods.)

In spite of these shortcomings, Connector/C++ is an interesting library for programmers who produce large quantities of C++ code. For such individuals, the steep learning curve is paid off with relatively elegant and, with some effort, quite readable code. However, those who write C++ code only occasionally would probably be better served by the C API described in the previous section (which is, of course, usable under C++).

Pointer

Connector/C++ is the official MySQL-supported API for C++ programming:

http://www.mysql.com/products/mysql++/index.html

In addition, however, there are several alternative C++ APIs. Links to those can be found at the following addresses:

http://www.mysql.com/portal/software/
http://www.mysql.com/doc/en/Contrib.html
Pointer

A further variant to Connector/C++ can be found at the following address. This API is derived from Connector/C++, but it is distinguished by a somewhat clearer programming style (with fewer syntax variants and alternative expressions). Unfortunately, there is no documentation whatsoever:

http://mysqlcppapi.sourceforge.net/

If you develop KDE or QT programs, the QT library offers, since version 3, classes for database access. In such cases it is useful to use these classes instead of Connector/C++. (However, the simultaneous use of QT and Connector/C++ is problematic due to name conflicts and conflicting modes of access.)

http://doc.trolltech.com/3.1/designer-manual-8.html

Assumptions

Connector/C++ is available for download on the MySQL server www.mysql.com for the current versions of Red Hat and SuSE Linux as a precompiled RPM package. To install it, you need only execute rpm -i packagename.rpm. The RPM packages use the following directories during installation:

Include files: /usr/include/sqlplus

Libraries: /usr/lib/libsqlplus.*

Documentation: /usr/share/doc/[packages/]Connector/C++

In addition to the Connector/C++ package, the libraries and include files from libmysqlclient must be installed; these were presented in the previous section. Both are components of the package MySQL-devel. The include files are usually found in the directory /usr/lib/mysql, and the libraries in /usr/lib/mysql.

The tested version 1.7.9 assumes libmysqlclient.so.10 (that is, the libmysqlclient version for MySQL 3.23.n). But it is expected that by the time this book appears, there will be precompiled packages that are compatible with libmysqlclient.so.12 (thus with MySQL 4.0).

If you are using a different operating system, then for you there exists Connector/C++ as a source-text package that you can compile yourself. If you are using gcc 3.2 as your compiler, you must first patch the code (an appropriate package is available for download). However, I have not tested this procedure, instead having tested complete RPMs under Red Hat 8.0 and SuSE 8.1.

Hello World in C++

As a first step into Connector/C++ programming, let us write a small program. It assumes that the database mylibrary is installed on the local computer (see Chapter 5). Furthermore, you must change the user name ("username") and password ("xxx") in the code for calling conn.connect. The code is documented to the extent that one can understand in principle what is going on. Details on the use of MySQL functions follow in the section on programming techniques:

// example file hellow/main.cpp
#include <iostream>
#include <sqlplus.hh> // Connector/C++ API
using namespace std;
int  main (int argc, char *argv[])
{
  Connection conn(use_exceptions);
  try {
    // create connection to the MySQL server
    conn.connect("mylibrary", "localhost", "username", "xxx");
    // execute SELECT query
            Query query = conn.query();
    query << "SELECT publID, publName \
          FROM publishers \
          ORDER BY publName";
    // store results
    Result result = query.store();
    cout << result.size() << " records found:" << endl << endl;
    // loop over all result records
    Row row;
    Result::iterator it;
    for(it = result.begin(); it != result.end(); it++) {
  row = *it;
  cout << "publId=" << row["publId"] << "   "
    << "publName=" << row["publName"] << endl; }
    return 0;
  } //  displays an error message if there were
    // problems with the connection or with the SELECT query
    catch(BadQuery er) {
    cerr << "connection or query exception: " << er.error << endl;
    return 1;
  }
}

Upon execution, the program yields the following result:

$  ./hellow
6  records found:
publId=1   publName=Addison-Wesley
publId=2   publName=apress
publId=9   publName=Bonnier Pocket
publId=5   publName=Hanser
publId=3   publName=New Riders
publId=4   publName=O'Reilly & Associates

Compiling and Linking Connector/C++ Programs

To compile and link the above introductory example under Linux or comparable Unix system, execute the following command. (The locations of the include files can be different, of course, on different systems.)

$  g++ -I/usr/include/mysql -I/usr/include/sqlplus  \
    -l sqlplu -lmysqlclient -o hellow main.cpp

As with C projects, it can happen that you will have to specify some additional link options for the libraries libz, libnsl, etc. With the additional option -Wno-deprecated you stop the C++ compiler from complaining about the obsolete (according to the current C++ standard) use of header files within the include files of Connector/C++.

If you wish to use make, the following Makefile provides a simple formula:

#  example file hellow-c++/Makefile
CPP = g++
INCLUDES = -I/usr/include/sqlplus -I/usr/include/mysql
LIBS =   -lsqlplus -lmysqlclient -lz
OPTS = -Wno-deprecated
all:hellow
main.o: main.cpp
    $(CPP) -c $(OPTS) $(INCLUDES) main.cpp
hellow: main.o
    $(CPP) -o hellow main.o $(LIBS)
clean:
    rm -f hellow main.o

Program Development with KDevelop

The development of Connector/C++ programs with KDevelop breaks down at two points:

  • KDevelop deactivates C++ exceptions by default. However, Connector/C++ makes extensive use of them.

  • KDevelop uses the options -pedantic and -ansi during compilation. The Connector/C++ include files cannot be thus compiled without errors.

Although the activation of exceptions is relatively simple (through the use of CXXFLAGS="$CXXFLAGS $USE_EXCEPTIONS" in configure.in.in; see the FAQs to KDevelop under http://www.kdevelop.org), the deactivation of the options -pedantic and -ansi requires deeper penetration into the nested configuration files of the automake system used by KDevelop.

Programming Techniques

Establishing the Connection

The connection to the MySQL server is established through a Connection object. If errors are to be trapped while the connection is being established, the object must be created with the constructor Connection(use_exceptions). If the connection fails to be established, then a BadQuery exception is triggered.

The actual connection is established with the method connect,to which up to four parameters may be passed: the database name, host name, user name, and password. Unspecified parameters are automatically read from my.cnf ([client] section). If no information is to be found at that location, then default values are used, namely, localhost as host name and the current login name as user name:

Connection conn(use_exceptions);
try {
    conn.connect("databasename", "hostname", "username", "password");
} // if an  error occurs
catch(BadQuery er) {
    cerr << "error is establishing the connection: " << er.error << endl;
    return 1;
}

Instead of using connect, you can create a connection with real_connect. The advantage is that real_connect supports some additional parameters (such as for the port number and timeout period).

Once the connection has been established, you can use the methods of the Connection object to create additonal objects (such as the Query object, to be defined shortly), determine various properties of the current connection with server_info, client_info, etc., execute SQL commands with exec, shut down the server with shutdown, etc.

To break the connection prematurely, execute the close method. (The connection is broken automatically at the end of the program.)

conn.close();

Executing Queries

To execute a query without results (thus not SELECT), you can simply pass the SQL character string to the method exec of the Connection object. If you are sending an INSERT command, you can also determine the ID number of the new data record with insert_id:

conn.exec("INSERT INTO publishers (publName) VALUES ('test')");
int newid = conn.insert_id();

Evaluating SELECT Queries

If you execute a SELECT command and wish to evaluate its result, then you should use objects of the classes Query, Result, and Row:

Query   query = conn.query();    // generae Query object
query << "SELECT  . . .    ";        // execute SQL command
Result result = query.store(); // store results

Once a Result object is at hand, you can determine the number of records returned by the SELECT command with one of the (equivalent) methods size or rows:

size_type rows = result.size();

If you wish to loop through all result records, then the Result object can provide an iterator. The following code lines show its use:

// loop over all result records
Row row;
Result::iterator it;
for(it = result.begin(); it != result.end(); it++) {
    row = *it;
    // evaluate row
}
Remark

Instead of Result,you can also use the class ResUse:

ResUse resuse = query.use();

The main difference here is that all the data are not immediately transferred to the client program; instead, they must be read one at a time in predetermined order via resuse.fetch_row( ). (Functionally, the classes Result and ResUse correspond to the functions mysql_store_result and mysql_use_result of the C API.)

Within Connector/C++, the class Result is derived from ResUse. Result offers some additional methods that enable, for example, navigation among the records held in memory.

You can determine the number of columns with the method size of the Row object. Access to the individual data fields is accomplished with row[0], row[1], etc., or in the more readable, but decidedly less efficient, manner row["column_name"].

Worthy of note is the return data type of row[ ... ]: It involves objects of the class RowData, which can be conveniently transformed by cast operations into the basic data types of C++ (for example, int, double, Date). On this point, Connector/C++ offers fundamental advantages over the C API, which always returns only character strings.

Thus, if the first column of a SELECT result has the data type DATE or TIMESTAMP, then you can assign the contents of this column to the variable mydate:

Date mydate = (Date)row[0];

Please note, however, that this transformation functions only if suitable data are available. (Thus you cannot change a Double into a date.)

Moreover, you should first test using row[ ... ].is_null( ) whether the field contains NULL. In such a case, the transformation can result in an error. (In some cases, NULL is simply transformed to 0 or 0000-00-00. But even in such cases, a NULL test should be made to distinguish 0 from NULL.)

If you no longer require a Result or ResUse object, you should release it with a call to purge. In particular, if you are working with ResUse objects, then the execution of purge is often explicitly required before the next SQL command can be executed. (Otherwise, an error occurs: commands out of sync.)

Using Binary Data

If you wish to store binary data in a MySQL table or to read stored data from a table, Connector/C++ offers special manipulators and methods. First, for storing data, thus for formulating an SQL command in which all special characters and zero bytes are coded with backslash combinations, you use the manipulator escape for an arbitrary iostream object. This has the effect that special characters in the immediately following data element are coded in accord with MySQL format. escape is particularly well suited if you are passing a query with << to a Query object or are composing an SQL command with << in an ostrstream object:

query << "INSERT ... VALUES('" << escape << binarydata << "')";

In reading data, you do not access binary data with a Row object (i.e., with row[column]), but instead, you use the method row.raw_data (column). You must first determine with the method fetch_lengths the extent of the data. This method returns a field with elements for all columns of the current record. fetch_lengths is available for objects of the classes Result and ResUse:

unsigned long *lengths;
lengths = result.fetch_lengths();

BLOB Example Program

The following example program stores the contents of the local file test.jpg in the table exceptions.test_blob (see Chapter 5). From there, the data are read and stored in the new file test-copy.jpg.Finally, the new record is deleted.

The transmission of data from the file to the database is somewhat complex, since the file first has to be read into a *char buffer and then copied into a string buffer; this string buffer is finally used to compose the SQL command in an ostrstream buffer. Nevertheless, the reading of data with raw_data is accomplished significantly more elegantly than with C code:

// example file blob/main.cpp
#include <iostream>
#include <fstream>
#include <sqlplus.hh>
using namespace std;
    int  main (int argc, char *argv[])
{
  Connection conn(use_exceptions);
  try {
    conn.connect("exceptions", "localhost", "username", "xxx");
    // first place binary data in a *char buffer and then
    // in a string buffer
    ifstream infile("test.jpg", ios::in | ios::binary | ios::ate);
    long size = infile.tellg();
    char *buffer = new char[size];
    infile.seekg(0, ios::beg);         // read from beginning of file
    infile.read(buffer, size);         // file --> *char
    string sbuffer(buffer, size);   // *char --> string
    infile.close();
    // store string buffer in the database
    ostrstream strbuf1;
    strbuf1 << "INSERT INTO test_blob (a_blob) VALUES ('"
<< escape << sbuffer << "')" << ends;
    conn.exec(strbuf1.str());
    // read BLOB field from the database
    int id = conn.insert_id();         // AUTO_INCREMENT id
        Query query = conn.query();
    query << "SELECT a_blob FROM test_blob WHERE id=" << id;
    Result result = query.store();
    Row row = result.fetch_row();
    unsigned long *lengths;
    lengths = result.fetch_lengths();
    // store data in a new file
    ofstream outfile("test-copy.jpg", ios::out | ios::binary);
    outfile.write(row.raw_data(0), lengths[0]);
    outfile.close();
    //release Result object, delete temporary table
    result.purge();
    ostrstream strbuf2;
    strbuf2 << "DELETE FROM test_blob WHERE id=" << id << ends;
    conn.exec(strbuf2.str());
    // close connection
    conn.close();
    return 0;
} //  catch exceptions
    catch(BadQuery er) {
    cerr << "error: " << er.error << endl;
    return 1;
  }
}

Error-Handling

As we have already mentioned in our description of the Connection class at the beginning of this section, error-handling in Connector/C++ is based on exceptions. For this type of error-handling to be activated, the first Connection object must be generated with the constructor Connection(use_exceptions). The result is that exceptions will be triggered that result from objects that are derived from this start object.

The most important exceptions that are triggered by operations within Connector/C++ are BadQuery (error in establishing the connection or in the execution of an SQL command), BadConversion (error in data type transformation), and BadNullConversion (error in data transformation due to data containing NULL). The following lines offer a simple plan for minimal error-handling in an entire Connector/C++ program:

int main(int argc, char *argv[])
{
  Connection conn(use_exceptions);
  try {
        . . .    all the code
  }
  catch(BadQuery er)           { . . . error-handling }
  catch(BadConversion er)      { . . . error-handling }
  catch(BadNullConversion er)  { . . . error-handling }
}

Specialized SQL Structures (SSQL)

In the previous examples, the individual columns were addressed with row[n] or row["name"]. Connector/C++ offers yet another possibility, using various sql_create macros, to create a structure at the start whose elements correspond to the columns of the query result. This has the advantage that the data appear immediately in the correct format, and so no cast operations are necessary.

The starting point for our first SSQL experiments is to consider the macros sql_create_basic_<n>, where n specifies the number of parameters. The macro call sql_create_basic_2 defines the structure for the following example program (two columns: publID and publName):

sql_create_basic_2(publisher,     // Name of the structure
               0,                 // number of columns to compare
               0,                 // number of columns to initialize
               int, publId,       // column publID
               string, publName);// column publName

With this macro call, publisher specifies the name of the structure being defined. The second parameter tells how many columns should be considered in making comparisons between objects of the structure. The third parameter specifies how many columns can be initialized by the constructor of the structure. The remaining parameters specify the data types and names of the the columns. As data type one may use MySQL-specific types that are known to Connector/C++ (such as Date, Time, and DateTime).

The following program returns, like the "Hello World" example of this section, an alphabetic list of all publishers with their publID numbers. New here is the Result method storein, with which the found records are copied into a vector of the structure publisher.Within the loop over elements of the vector, the individual column elements can now be conveniently and efficiently addressed in the manner in->name:

// example file ssql/main.cpp
#include <iostream>
#include <vector>
#include <sqlplus.hh>
#include <custom.hh>
using namespace std;
sql_create_basic_2(publisher. . .); // defines the structure  publisher;
                                 // for details, see above
int  main (int argc, char *argv[])
{
  Connection conn(use_exceptions);
  try {
    // create connection to MySQL
    conn.connect("mylibrary", "localhost", "root", "");
    // process SELECT query
              Query query = conn.query();
    query << "SELECT publID, publName \
          FROM publishers \
          ORDER BY publName";
    // copy result in vector of the the data type publisher
    vector <publisher> result;
    query.storein(result);
    // loop over all data records
    vector <publisher>::iterator it;
    for(it=result.begin(); it!=result.end(); it++) {
  cout << "publID=" << it->publId << "   "
        << "publName=" << it->publName
        << endl;
    }
  } //  error-handling
    catch(BadQuery er) {
    cerr << "error: " << er.error << endl;
    return 1;
  }
}
Pointer

The Connector/C++ documentation describes a host of additional syntax variants and application options beyond this simple example. However, there is no room here to discuss them.


Chapter 17: Visual Basic, C#, ODBC

Overview

ODBC (open database connectivity) is above all a widely used interface in the world of Windows for accessing database systems. Thanks to the driver Connector/ODBC, MySQL can be accessed via ODBC.

This chapter begins with some background information about ODBC and the setting up of data source names (DSNs), which simplify access to ODBC data sources (for example, a MySQL database). The following two sections show how you can now use Access and Excel for processing and analyzing MySQL databases.

The main part of this chapter concerns itself with the programming of MySQL applications on the basis of the database libraries ADO and ADO.NET. This will also involve the programming languages VB (Visual Basic), VBA, VB.NET, and C#. (Under Visual Basic 6, you can use the library MyVbQL.dll instead of ADO and ODBC, which is more efficient in many cases. This special case will be handled as well.)

For those whom this chapter has convinced of the advantages of MySQSL, the last section describes the program mssql2mysql, a database converter that transfers databases from the Microsoft SQL server to MySQL.


Fundamentals

ODBC (Open Database Connectivity) is a popular mechanism under Windows for standardized access to a great variety of database systems. The only condition on ODBC for it to be used to access MySQL is the installation of Connector/ODBC, the ODBC driver for MySQL.

Pointer

This chapter is based on the assumption that you have installed Connector/ODBC as described in Chapter 2. There is extensive documentation for Connector/ODBC, but it is scattered over three separate places, not all of which are up to date. Basic information is available in the MySQL documentation. There is also an extensive "frequently asked questions" (FAQs) section, as well as a technical reference. These documents were last available at

http://www.mysql.com/doc/en/ODBC.html
http://www.mysql.com/products/myodbc/faq_toc.html
http://www.mysql.com/products/myodbc/manual_toc.html

There is also a Connector/ODBC mailing list, which can be found at http://lists.mysql.com/.

Determining the Connector/ODBC Version Number

To determine whether Connector/ODBC is installed on your computer (and in what version), execute START |SETTINGS |CONTROL PANEL |ADMINISTRATIVE TOOLS |DATA SOURCES (ODBC) and search in the DRIVERS dialog sheet of the ODBC dialog for the Connector/ODBC driver (see Figure 17-1).

Click To expand
Figure 17-1: Determining the Connector/ODBC version number
Remark

Note that this entire chapter is based on Connector/ODBC 3.51.n. (I ran my tests using versions 3.51.05 and 3.51.06.) Version 3.51.n offers a number of substantial improvements over the previous version 2.50.n. Make sure that youare not inadvertently working with an obsolete version.

A Small Glossary from the World of Microsoft Databases

If you are new to the world of Microsoft databases, then ODBC is not the only abbreviation that is going to confuse you. Here is a brief overview:

  • ODBC (Open Database Connectivity): ODBC is a popular, primarily Windows, mechanism for integrated access to a variety of database systems. ODBC is rather old, but it is supported by countless old and new database systems and libraries and will therefore likely be the most important producer-independent standard for communication and programming with database systems under Windows for a long time to come. ODBC is also available under Unix, but is seldom used in that context.

  • OLE-DB: OLE-DB is the successor to ODBC, thus again an interface for providing unified access to various database systems. For MySQL there is at present, sad to say, no mature OLE-DB driver. However, there is an ODBC interface for OLE-DB, by means of which all ODBC-compatible databases without their own OLE-DB drivers can be addressed. As a MySQL programmer you generally have nothing to do directly with OLE-DB, since you use ADO for the actual programming.

  • ADO (ActiveX Data Objects): ADO is a collection of objects that are used in the programming languages Visual Basic, VBA, Delphi, etc., for creating database programs. ADO stands between the programming language and OLE-DB. For programmers, the object-oriented ADO library offers much more convenience than direct access to the database via OLE-DB functions. Later in the chapter, we provide information on ADO programming with Visual Basic for accessing MySQL databases.

  • ADO.NET: ADO.NET is the database interface of the new Microsoft .NET Framework. (This framework is an enormous class library that can be used only by .NET-compatible programming languages such as Visual Basic .NET and C#.) ADO.NET has a name similar to that of ADO, but internally it has little in common with it; the two are completely incompatible.

  • MDAC (Microsoft Data Access Components): MDAC is the general term for the countless Microsoft database components and libraries (including ADO and OLE-DB). Various MDAC versions are installed as parts of Windows, Office, Internet Explorer, etc. The most current versions can be found (at no cost) at the following address:

    http://www.microsoft.com/data/download.htm

  • DAO, RDO: DAO and RDO are similar to the ADO libraries for database programming. These are precursors to ADO and are now supported by Microsoft only half-heartedly. DAO and RDO are not discussed further in this book.

  • DSN (Data Source Name): With many programs with an ODBC interface (e.g., Access, Excel) you cannot directly select a database. Instead, you must have previously defined a DSN for the database in the ODBC manager. Access to the database then proceeds via the DSN. (If you access MySQL via a programming language, you can create the database access without a DSN as well. Later on, we will show both variants, that is, access with and without a DSN.)

Setting Up a DSN for a MySQL Database

DSNs are usually set up in the ODBC administrator dialog. (Many Windows programs themselves provide dialogs for this purpose, but they are no improvement.) You launch the ODBC administrator dialog in the system control, under Windows 2000, for example, with START |SETTINGS |CONTROL PANEL |ADMINISTRATIVE TOOLS |DATA SOURCES (ODBC). There you have the choice among three DSN types: user, system, and file DSNs.

  • User DSNs: These are available only to the user who has defined the DSN. (During Connector/ODBC installation a user DSN with the name sample-MySQL is automatically set up.) However, with this DSN the correct database name must first be employed before the DSN can be used.

  • System DSNs: These DSNs are available to all users of the computer. With both user and system DSNs the settings are managed internally by ODBC.

  • File DSNs: Here we are dealing with individual files with the identifier *.dsn, in which all setting data are stored in text format. These files are usually stored in the directory Program Files\Common Files\ODBC\Data Sources. File DSNs are more transparent to some extent, because it is clear what information is stored where.

Tip

If you wish to access more than one MySQL database, you must make use of the corresponding number of DSNs.

The definition of a new DSN begins with clicking on the ADD button in the User DSN or System DSN dialog sheet. There appears a dialog in which you can choose from among various ODBC drivers.

The correct choice in our case is, of course, MySQL. A double click now leads into a dialog (see Figure 17-2), in which you must make the following settings:

  • Data Source Name: Here is where you specify the name of the DSN (that is, under what name the data source can be addressed by ODBC programs). Select a name that is connected to the database name and the use to which the DSN is to be put.

    Click To expand
    Figure 17-2: Basic Connector/ODBC setting

  • Host/Server Name (or IP): Here you give the name or IP number of the computer on which MySQL (i.e., the server) is running. If it is the local computer, then give localhost.

  • Database Name: Here you give the name of the MySQL database (for example, mylibrary, if you wish to access the database used for many examples in this book).

  • User and Password: Of course, the access system described in Chapter 9 holds for Connector/ODBC as well. (If you have begun to read the book at this point and are unable to achieve a Connector/ODBC connection, then inadequate access privileges is the most likely cause of the difficulty.)

    In the fields User and Password you can input the user name and password for access to the database. However, whether you should actually do this is another question entirely. First, you provide access to the database to anyone who can use the DSN. Second, the password is not encrypted and can therefore easily be discovered.

    To circumvent this security risk, you can leave both fields empty. Then every time you or someone else wishes to use this DSN to obtain a connection to MySQL, the dialog pictured in Figure 17-2 appears. While this is very practical, in that name and password can be input, it is very confusing for the average user. It is not clear that user name and password are indeed to be entered, and it is possible that the poor user will stumble around in the other input fields of this dialog, which can lead to errors that will be difficult to unravel.

  • Port: Here is where the IP port can be specified, in case it should be other than 3306. In most cases this field is left empty.

    If MySQL is running on a computer other than the one on which Connector/ODBC is running and there is a firewall between the two computers, then port 3306 must be opened. Otherwise, no communication between MySQL and Connector/ODBC will be possible.

  • SQL Command on Connect: Here an SQL command can be given that is to be executed immediately upon establishment of the connection. In rare cases this can be used to create a temporary table or set a variable (SET). As a rule, this field is left empty.

With the button TEST DATA SOURCE you can quickly check whether a connection to the MySQL server succeeds with the current settings.

If you later change the settings of a DSN, simply open the ODBC administration dialog. A double click on the DSN (or the button CONFIGURE) leads again to the dialog shown in Figure 17-2.

Connector/ODBC Options

The button OPTIONS leads to a second dialog box with a number of options (see Figure 17-3). The following list describes the most important options, where the order reflects that in the OPTIONS dialog box. The number in parentheses represents the numerical value of the option, which is important if you wish to set the options in program code (as we shall do in the upcoming ADO example).

Click To expand
Figure 17-3: Connector/ODBC Options

  • Don't Optimize Column Width (1): If the query SELECT col FROM table returns character strings with a maximal length of n characters, and n is smaller than the maximal string length for this column, then Connector/ODBC returns n as the optimal column width. The option Don't Optimize Column Width avoids such problems.

  • Return Matching Rows (2): With many commands (e.g., UPDATE), the MySQL server can return the number of affected data records (affected rows). However, many ODBC clients are incapable of interpreting this information correctly. If this option is used, Connector/ODBC returns found rows instead of affected rows (thus 0 in the case of UPDATE commands). This option must be set for Connector/ODBC to be able to function in combination with a number of programs (including Microsoft Access and Visual Basic/VBA with ADO).

  • Allow Big Results (8): With this option, the size of packets that are transmitted between an ODBC program and the server is unlimited. This option must be used if large SELECT commands are anticipated or large BLOB fields are to be processed.

  • Use Compressed Protocol (2048): Data transmission between the ODBC program and MySQL is in compressed form, thus reducing the amount of network traffic. However, the cost is more CPU time on both client and server, since all data must be compressed and decompressed.

  • Change BIGINT Columns to INT (16384): Most Microsoft libraries are unable to deal with 64-bit integers. This option has the effect that MySQL BIGINT fields are automatically reduced to 32 bits upon transmittal. This option is necessary in particular for ADO programming with Visual Basic/VBA. Be thou warned, however: The most-significant 32 bits will be lost.

  • Safety (131072): This option has the effect that Connector/ODBC makes some additional checks. This option should be used when an ODBC client experiences problems in the processing of MySQL data. (The documentation does not offer a more precise description.)

  • Don't Prompt on Connect (16): This option prevents the dialog displayed in Figure 17-2 from being displayed during the establishment of a connection due to the lack of relevant information. (Even if this option is activated, the dialog will appear if the database login is unsuccessful on account of an incorrect or missing user name or password.)

  • Enable Dynamic Cursor (32): With this option, Connector/ODBC supports the cursor type Dynamic. The cursor is made available by the ODBC driver, not directly by the MySQL server (which currently supports no cursors at all). By default, the Dynamic option is deactivated for the sake of efficiency. Note that this option works only since Connector/ODBC 3.52.06 and is not available in versions 2.50.n.

  • Don'tCache Results (1048576): Normally, all results of a SELECT command are transmitted as a block from server to client. This option prevents that. It makes sense only when huge (millions of records) SELECT results are being processed. This option takes effect only if in ADO programming a forward only cursor is used.

  • Force Use of Named Pipes (8192): This option results in named pipes (instead of TCP/IP) being used in communication between client and server. This is possible only under Windows NT/2000/XP, and only if you use a MySQL server that was compiled with named pipes support (which is not the default behavior).

In many cases, it suffices to activate only the options Don't Optimize Column Width and Return Matching Rows.

Pointer

Further information on Connector/ODBC can be found at http://www.mysql.com/products/myodbc/faq_3.html.


Access

Introduction

Access and MySQL are two programs that are fundamentally different from each other. Access has a very well developed user interface, which simplifies the design of databases as well as the development of database programs. Unfortunately, Access is as slow as molasses outdoors in Moscow in winter if more than three or four users wish to access the database simultaneously. Furthermore, Access is extremely unsuited for Internet applications.

In comparison, MySQL is incomparably more efficient and secure in multiuser operation. Here the problem is in using the darn thing: Although there exists a variety of user interfaces to MySQL, at present, none of these offers the convenience of Access.

Taking all of this into consideration, it seems a good idea to try to combine the best of both worlds. Even though there are many restrictions and limitations, the potential applications of Access in combination with MySQL are quite numerous:

  • Access tables can be exported to MySQL and there processed further.

  • MySQL tables can be imported to Access or a link to the tables can be created and the data further processed.

  • Data from Access and MySQL databases can be combined.

  • You can develop the design of a new database in Access and then export the entire database to MySQL.

  • You can use Access as an interface for changing the contents of MySQL tables.

  • You can develop complex queries in Access and then place the SQL code in your scripts.

  • You can create and print database reports.

  • You can develop VBA code in Access for automatically processing data from MySQL databases in Access.

  • You can create diagrams of database structures. (All such diagrams in this book were created in this manner.)

Remark

This section assumes that you are familiar with Access. An introduction to Access would be beyond the scope of this book. All procedures described in this section were tested with Access 2000.

Problems

Don't let your hopes soar too high. Access was never conceived as an interface to MySQL and will never become one. Thus it is not surprising that a number of difficulties exist in stretching Access on the Procrustean bed of an interface to MySQL:

  • There are several data types that are available either only in Access or only in MySQL. In import/export such data types can create trouble. For example, MySQL ENUMs and SETs are turned into simple text fields upon importation. There are also problems to be had with single-precision FLOATs. In general, you should use DOUBLE floating-point numbers if you plan on working with Access.

  • You should generally count on having to change various attributes of a MySQL table as a result of importing it into Access and later exporting it back to MySQL.

  • MySQL presently offers a way of creating links between tables only for the InnoDB format. Relations (including rules for ensuring referential integrity) that are easily constructed in Access are lost upon exportation to MySQL.

  • The SQL dialect of MySQL and that of Access are not quite identical. Thus it can happen that SQL queries developed in Access will function in MySQL only after some minor adjustments.

Assumptions

  • If you wish to change data from linked MySQL tables in Access, the option Return Matching Rows must be set in the definition of the DSN. If you are working with Access 2.0, you will also need the option Simulate ODBC 1.0.

  • Furthermore, all linked MySQL tables must be equipped with a TIMESTAMP field so that data can be altered. (You don't need to concern yourself with the care and feeding of this field, since both MySQL and Access automatically store the time of the last change. Access requires this information to distinguish altered data from unaltered data.)

  • All MySQL tables must be equipped with a primary index (usually an INT field with the property AUTO_INCREMENT).

  • A significant source of problems is the existence of incompatible data types:

    • MySQL tables use DOUBLE instead of FLOAT and DATETIME instead of DATE.

    • Avoid BIGINT, SET,and ENUM.

    • Access has occasional problems with BLOBs and believes that it is dealing with OLE objects. It is usually better to represent such columns in MySQL as BLOBs.

    Tip

    If you develop queries in Access and then wish to use the SQL code in MySQL, you will frequently find yourself in a compatibility quagmire. (The SQL dialect of Access is different from that in MySQL.) Here is how you can solve this problem: Work with linked tables, and in Access execute the following command (while the window for query formulation is open): QUERY |SQL SPECIFIC |PASS-THROUGH. Youthereby let Access know in no uncertain terms that the program should hold more rigorously to SQL standards.

    Tip

    If you are working in Access with linked tables andifafter an alteration in a data record you see #deleted displayed, then one of the above conditions has not been satisfied.

Importing and Exporting Tables

Imported Tables Versus Linked Tables

If you wish to process MySQL data in Access, then you have two possibilities: You can first import the table(s) into Access and then do the processing, or else you can merely create a link between Access and the MySQL tables.

  • With linked tables you can insert or alter data, although the table will then be managed by MySQL. Here Access serves simply as the interface. A change in the table's properties (insertion or deletion of a column, for example) is not possible with linked tables. The principal advantages are that the creation of links proceeds very quickly (independent of the size of the tables) and the data remain with MySQL (where they are well cared for).

  • There are no restrictions on imported tables as they pertain to changes in the table design. Since the tables are now stored in an Access database file, you can carry out all the operations provided by Access. However, there can be problems if you later attempt to return the table(s) to MySQL, since not all of the Access data types can be transformed to MySQL data types without further ado.

Importing MySQL Tables into Access

The first step is either to open an existing Access database or to create a new (empty) database. Then you execute FILE |GET EXTERNAL DATA |IMPORT and in the file selection dialog you choose ODBC as file type. As if by magic, the dialog SELECT

DATASOURCE appears. In the dialog sheet MACHINE DATA SOURCES, all the user and system DSNs are enumerated. Select the DSN of your choice. As a rule, there will now appear the dialog shown in Figure 17-2, in which you have merely to provide the user name and password.

If the connection to MySQL can be established, then the next dialog (see Figure 17-4) shows a list of all tables in the database specified by the DSN. There you can mark several tables at once (mouse button and Ctrl). The importation is initiated with OK, which can take a bit of time if the tables are large. (Access creates a local copy of the tables and stores them in an Access database file.)

Note that on being imported, many column attributes (e.g., AUTO_INCREMENT) are lost, and that the forms of indexes are frequently changed (so that a MySQL UNIQUE index in Access becomes an index that allows duplicates).

Click To expand
Figure 17-4: Selection of the tables to be imported into Access

Creating Links to MySQL Tables

The way to create a database link is the same as with the importation of a MySQL table. The only difference is that this time, you begin with the Access menu command FILE |GET EXTERNAL DATA |IMPORT |LINK TABLES.

Caution

If you change the properties of a table in MySQL (for example, if you add a new column), then you must update the connection to this table in Access. To do this, execute TOOLS |DATABASE UTILITIES |LINKED TABLE MANAGER and select the table to be updated.

Exporting Access Tables to MySQL

If you have imported a table from MySQL to Access and then changed it, or if you have developed an entire database in Access and then wish to transport it to MySQL, then you have to go in the opposite direction.

To do this, you mark the table in the Access database window, execute the command FILE |EXPORT, and choose the file type ODBC. In the EXPORT dialog you can specify the name of the table under MySQL. (As a rule, you can use the name being used under Access simply with an OK.) Now comes the familiar selection of the DSN. And that does it!

In principle, exportation functions acceptably well, but usually there is some additional manual work required. The following list gives some of the basic problems with exportation:

  • All indexes defined in Access are lost (including the primary index).

  • The Access column property Required is not translated into the MySQL attribute NOT NULL.

  • The Access column property Autonumber is not translated into the MySQL attribute AUTO_INCREMENT.

  • The translation of data types is not always optimal. For example, Currency is translated to DOUBLE (MySQL). Here DECIMAL would be preferable. In general, you should check your MySQL tables carefully for such losses in translation.

Unfortunately, there is no possibility in Access to export several tables simultaneously. Thus if you wish to export an entire database, you must repeat the steps outlined above for each table.

Converter: Access ® MySQL (exportsql.txt)

Due to the problems enumerated in the previous section that arise in the export of individual tables from Access to MySQL, Pedro Freire has programmed a converter that writes an entire Access database to a file *.sql, which then can be input with mysql. The quality of this converter is considerably better than ODBC Export, which is integrated into Access. This free program, exportSQL.txt, can be obtained at the following address:

http://www.cynergi.net/exportsql/

  • The method for exporting is documented in the program code. Here is a summary:

  • Load the database to be exported into Access.

    With Alt+F11 switch into the VBA editor, there insert a new module (INSERT |MODULE), and copy the entire file exportsql.txt into the module (see the link above).

  • You can work in the VBA editor only if you are working with an Access 2000 database. If the database was created with an earlier version of Access, then you must open a new module in the database window and insert the code into this module.

  • Change the export options in the program code. This involves a block of constants (Private Const name = ... ) that control certain parameters of the exportation process. As a rule, you will have to change only the two constants ADD_SQL_FILE and DEL_SQL_FILE, which specify the names of the file into which the export files are to be written.

  • Begin exporting with F5. (In the macro dialog you must select the procedure exportSQL.)

  • During the exportation process you may see displayed some warning messages relating to incompatibilities between Access and MySQL data types, which you must approve by clicking OK. The warnings will appear as well in the resulting files. (The affected lines begin with #Warning.)

  • In MySQL generate an empty new database (with CREATE DATABASE).

  • Execute the SQL command specified in esql_add.txt. You can best accomplish this with mysql:

    > mysql -u root -p databasename < q:\tmp\esql_add.txt
Enter password: xxxxx

If errors occur, you may have to alter esql_add.txt in a text editor. In some situations some small changes in the program code of the converter may be necessary.

Before making a new attempt, you must delete any existing data. For this, the second file esql_del.txt will be of help:

> mysql -u root -p databasename < q:\tmp\esql_del.txt
Enter password: xxxxx

Problems

  • Access permits a number of special characters in table and column names that MySQL does not allow. The following changes in the program code (indicated in boldface) will permit exportSQL.txt to replace these characters automatically with an underscore ( _ ):

    Private Function conv_name(strname As String) As String
...
Select Case Mid$(str, I, 1)
  Case " ", Chr$(9), Chr$(10), Chr$(13), "-", ")", "("
...

  • In those countries where floating-point numbers are formatted with a comma for the decimal point (MySQL is clueless about this), the following remedy is available:

    Sub exportSQL()
...
Select Case crs.Fields(cfieldix).Type
...
Case Else
  sqlcode = sqlcode & conv_str(str(crs.Fields(cfieldix).Value))

  • PRIMARY KEY columns must have the attribute NOT NULL in MySQL. However, exportsql.txt does not guarantee this condition, and that can lead to problems. The solution is to edit the column definitions of the export file esql_add.txt in a text editor.

Other Converters Between MySQL and Access

In addition to exportsql.txt, there are many other converters between MySQL and Access (in both directions) that I have not tested in detail, such as importsql and MyAccess. Many of these tools are freely available, while others are commercial programs. An overview of such programs is available at


Data Analysis with Excel

Excel is of interest to users of MySQL primarily as a tool for data analysis. Thanks to Connector/ODBC, you can import MySQL data into an Excel worksheet and there perform analyses, create graphics, and so on. In Figure 17-5 you can see, as an example, a pivot table that for the mylibrary database tells which publisher has published how many books in a given category. (Because of the relatively small number of data records the result is, of course, not very informative. But if mylibrary contained a large number of books, then this table would allow you to determine easily which publishers specialize in which subjects.)

Click To expand
Figure 17-5: Excel pivot table
Pointer

If you use the macro programming language VBA, which is included in Excel, then a broader range of options is open to you, particularly if your goal is to automate or simplify the steps discussed here. A brief introduction to database access via VBA (or Visual Basic) will be given later in the chapter.

Importing Data with MS Query

Whether you wish to insert MySQL data directly into an Excel worksheet (command DATA |GET EXTERNAL DATA |NEW DATABASE QUERY) or to create a pivot table or chart based on external data, Excel will launch the auxiliary program MS Query. This program functions as the interface between Excel and an external database, and it assists in the setting of import and query parameters.

In this program you select the DSN in the dialog CHOOSE DATA SOURCE. At that point, the QUERY WIZARD appears, which assists in the creation of a database query. In the first step you select the required tables or table fields (see Figure 17-6). In the next two steps you can specify the filter criteria (corresponding to WHERE conditions) and the sort order.

Click To expand
Figure 17-6: Selecting the desired columns in MS Query

If these specifications suffice for executing the query, you can now terminate MS Query with FINISH. Usually, however, it is preferable first to click on the option VIEW

DATA OR EDIT QUERY. In this case, the query is displayed by MS Query in development mode, where it can be optimized (see Figure 17-7). In particular, any relations between the tables that MS Query has not recognized on its own can be established. To accomplish this you simply Drag&Drop the ID fields from one table to the other. A dialog appears in which you can set the relational properties.

Click To expand
Figure 17-7: MS Query in development mode
Tip

Excel frequently has difficulties with the importation of information about data. You can solve some of these problems with the help of the SQL function CONCAT. To format a column with this function, open the dialog EDIT COLUMN in MS Query with a double click on the column in question. In the input field FIELD you set the expression present there in CONCAT( ... ). Forexample, from table.birthdate you would produce CONCAT(table.birthdate).


ADO Programming and Visual Basic/VBA

Introduction

In this section we describe access to MySQL databases via program code, where we use Visual Basic or VBA as our programming language. We use ADO as our database library. The actual access to the database is effected by means of Connector/ODBC, where between ADO and Connector/ODBC we have OLE-DB and the ODBC driver for OLE-DB. Thus communication takes place in the following way: VB/VBA ® ADO ® OLE-DB ® ODBC driver for OLE-DB ® Connector/ODBC ® MySQL. If MySQL is running on a computer as a Visual Basic program, then the first five stages of this communication chain take place on the client computer. It is Connector/ODBC that first communicates over the network with the MySQL server.

Pointer

In this book we are assuming that the reader has basic knowledge of ADO. If you know nothing much about ADO, do not know what Connection and Recordset objects are or the purpose of bound controls, then you should consult the literature on ADO programming before you read further, since we are not making much ado about nothing.

The code presented here was developed with Connector/ODBC 3.51.05, Visual Basic 6, and ADO 2.5. The code should function as well with VBA 6 and ADO 2.1 (Office 2000), except as it involves ADO controls.

Of course, you can also work with a scripting variant of Visual Basic, for example, if you wish to develop Active Server Pages (ASP) or programs for the Windows Scripting Host (WSH). In that case you must leave out As typename in the variable declaration and create ADO objects according to Set conn = CreateObject("ADODB.Connection"). Thereare several good articles specifically on ASP/MySQL programming available on the Internet, including the following:

http://www.devarticles.com/art/1/50
http://www.dwam.net/mysql/asp_myodbc.asp
Pointer

There is also an OLE-DB driver for MySQL (MyOLEDB). However, according to information on the Internet, the currently available version, 3.0, is not really ready for use, and moreover, further development has been halted. I have not tested this driver. If you wish to experiment with MyOLEDB, further information can be found on the Internet, in particular at the following location:

http://www.mysql.com/portal/sites/index.html
Pointer

Please note as well that MySQL may not be deployed in all circumstances without cost, despite popular opinion to the contrary. Free use is permitted only for open-source projects and for databases on dynamic web sites. If, on the other hand, you wish to develop, say, a commercial bookkeeping program based on MySQL, you must purchase a MySQL license for your customers. This issue is discussed in Chapter 1 and on the Internet, at

http://www.mysql.com/doc/L/i/Licensing_and_Support.html

Assumptions and Limitations

  • MySQL tables: Tables that you wish to alter via ADO Recordsets or in bound controls (Insert, Update, Delete) must have a primary index and a TIMESTAMP column.

  • Connector/ODBC options: In the configuration of a DSN, the options Don't optimize column width, Return matching rows, and Change BIGINT into INT must be selected. If the connection is made without a DSN, then instead, the equivalent setting Options=16387 (1 + 2 + 16384) must be specified. If you wish to use a dynamic server cursor in your ADO program (CursorType=adOpenDynamic, CursorLocation=adUseServer), then the correct Options setting is 16419 (1+2+32+16384).

  • Changing data with Recordset s and bound controls: If the above conditions have been satisfied, you can use Recordsets and bound controls to read, display, and alter data.

    This last point can often cause problems. Storing altered data (whether via a Recordset object or a bound control) functions properly in many simple cases, but in more complex applications it often leads to problems that are difficult to resolve. These problems do not always have to do with MySQL or Connector/ODBC, but with the vagaries of OLE-DB/ADO. Many experienced ADO programmers therefore do without bound controls, even if a Microsoft database system is used as the data source.

    For this reason you will continually find in MySQL newsgroups and mailing lists the recommendation to use Recordsets and bound controls exclusively for reading or displaying data. If you wish to alter data, then you should formulate a traditional SQL command (UPDATE ... ) and execute it with conn.Execute. Even if I have demonstrated both methods in this chapter (both the convenient editing of data with a Recordset object and the variant with SQL commands), I recommend the second variant.

  • Speed: Because of the many layers and libraries between Visual Basic and MySQL, and also because of the lower speed in the processing of character strings with Visual Basic, solutions using Visual Basic and MySQL tend to be comparatively slow. Better performance is promised by the MyVbQL library, which establishes a direct connection between Visual Basic and MySQL (without ODBC).

Example Programs

As with the other chapters in this book, there are extensive examples available at www.apress.com. These examples assume that you have Visual Basic 6. Most of the examples also assume previous installation of the mylibraryodbc database (this database corresponds to the mylibrary database used throughout this book, except that in this version, all tables have been equipped with an additional timestamp column).

Note that at the beginning of the program (usually in Form_Load), you must set the connection parameters (computer name, user name, password) in accord with the settings of your system. In the example programs, the computer name is localhost, the user name is root, and the password is saturn.

Establishing the Connection

Establishing the Connection with a DSN

The following lines show a code outline for establishing a connection to MySQL based on a DSN and for executing a simple SELECT query. The resulting record list can then be processed with the properties and methods of the rec object:

Dim conn As Connection
Dim rec As Recordset
Set conn = New Connection
conn.ConnectionString = "DSN=mysql-mylibrary"
conn.Open
Set rec = New Recordset
rec.CursorLocation=adUseClient
rec.Open "SELECT * FROM tablename", conn
. . . 'process record list
rec.Close 'close record list
conn.Close 'terminate connection

The only really interesting code line is the one for setting the ConnectionString property. In the simplest case you specify the DSN. Optionally, you can also specify the user name and password. These parameters must be separated by semicolons. The ConnectionString character string looks like this:

conn.ConnectionString = "DSN=mysql-mylibrary;UID=root;PWD=xxx"

The UID and PWD settings have precendence over the DSN settings. Here are the ConnectionString parameters for establishing a connection with a DSN:

DSN=name | DSN
UID=name | user name for the MySQL connection
PWD=password | password for the MySQL connection

Establishing a Connection Without a DSN

With Visual Basic you can also establish a connection to a MySQL database without previously having defined a DSN. (This is particularly practical if you wish to develop a program to access an arbitrary MySQL database.) The ConnectionString character string looks like this:

conn.ConnectionString = "Provider=MSDASQL;" + _
  "DRIVER=MySQL ODBC 3.51 Driver;" + _
  "Server=localhost;UID=username;PWD=xxx;" + _
  "database=databasename;Option=16387"

Here are the ConnectionString parameters for establishing a connection without a DSN:

Provider=MSDASQL | name of the ODBC driver for OLE-DB (Microsoft Data Access SQL)
Driver={MySQL ODBC 3.51 Driver} | name of the MySQLODBCdriver
Server=name | name or IP address of the computer on which MySQL isrunning
Port=n | IP port of the MySQL server (default 3306)
UID=name | user name for the MySQL connection
PWD=password | password for the MySQL connection
Database=name | name of the database
Prompt=noprompt/complete | display Connector/ODBC dialog if connection fails
Option=n | Connector/ODBC options

Prompt=complete leads to the program automatically displaying the dialog depicted in Figure 17-2. This dialog can be used, for example, to provide a missing or incorrect password. Moreover, an arbitrary character string must be specified in the field WINDOWS DSN NAME so that the dialog can be exited.

Since the dialog is not particularly user-friendly (the many choices are more confusing than helpful), it is better to do without Prompt=complete. (The default setting is noprompt.)

Instead, execute On Error Resume Next before establishing the connection and test after establishing the connection whether an error has occurred. If this is the case, display your own dialog, in which the user can input other connection parameters as necessary and from which the program can then construct a new ConnectionString character string.

Options enables the setting of a number of Connector/ODBC connection options. The required value is determined from the sum of the values of the individual options. For Visual Basic/ADO programs, the value 16387 = 1 + 2 + 16384 must generally be specified, corresponding to the options Don't optimize column width, Return matching rows, and Change BIGINT into INT. If you wish to use dynamic cursors in your ADO program, the correct Options setting is 16419 =1+2+32+ 16384.

Remark

If you are working with the old Connector/ODBC version 2.50.n (and not with the current version 3.51.n), you must use the setting Driver=MySQL in ConnectionString. Note, though, that a number of functions are unavailable in version 2.50.n.

Establishing the Connection with DataEnvironment

If you are working with Visual Basic Professional or Enterprise version 6, then you have DataEnvironment available for setting connection properties.

In the properties dialog for the Connection object, in the dialog sheet PROVIDER, the entry MICROSOFT OLE-DB PROVIDER FOR ODBC DRIVERS. You can then choose, in the dialog sheet CONNECTION (see Figure 17-8), an already defined DSN or assemble an equivalent character string with the required connection properties with the button BUILD. (This button leads to the familiar ODBC dialogs and finally to the Connector/ODBC dialog depicted in Figure 17-2. However, the result of the setting is copied as a character string into the DataEnvironment dialog, where it can be edited as necessary.)

Click To expand
Figure 17-8: DataEnvironment dialog for setting the connection properties

In the further program code you can now access the Connection object. (The following lines assume that the DataEnvironment object has the name DE and the Connection the name Conn.)

Dim rec As Recordset
Set rec = New Recordset If DE.Conn.State = adStateClosed Then
  DE.Conn.Open
End If
rec.CursorLocation=adUseClient
rec.Open "SELECT * FROM database", DE.Conn
...

The DataEnvironment helps not only in setting the connection data, but also in the development of SQL queries. With the SQL generator you have a convenient tool for this purpose. You can then access the query in code via the Command object.

Connection Properties

The ConnectionString parameters described in this section represent only a small part of the available parameters. For those parameters that are not specified, ADO/ODBC simply uses various default settings.

Sometimes, it is useful to know how these parameters are set. The property ConnectionString, which is changed in establishing the connection (and then no longer contains the character string originally set), provides our first overview. Here most of the parameters are collected into a sort of superparameter called Extended Properties:

conn.ConnectionString = "Provider=MSDASQL;Driver=MySQL;Server=  . . .    "

conn.Open
Debug.Print conn.ConnectionString
    Provider=MSDASQL.1;Extended Properties=
      "DRIVER={MySQL ODBC 3.51 Driver};DESC=;DATABASE=mylibraryodbc;
        SERVER=localhost;UID=root;PASSWORD=saturn;PORT=;
        OPTION=16387;STMT=;"

Considerably more information is provided by the Properties enumeration of the Connection object. Moreover, the values saved there can be more easily extracted:

Debug.Print conn.Properties("Max Columns in Index")
  32

A complete list of all properties and their values can be obtained with the following loop:

Dim p As Property
For Each p In conn.Properties
    Debug.Print p.Name & " = " &  p.Value
Next

And here is the output (squeezed into two columns to save space and sorted for easier reference):

Accessible Procedures = False                Catalog Usage = 29
Accessible Tables = True                     Column Definition = 1
Active Sessions = 0                          Connect Timeout = 15
Active Statements = 0                        Connection Status = 1
Asynchable Abort = False                     Current Catalog = mylibraryodbc
Asynchable Commit = False                    Data Source =
Autocommit Isolation Levels                  Data Source Name =
= 4096                                       Data Source Object Threading
Catalog Location = 1                         Model = 1
Catalog Term = database                      DBMS Name = MySQL
DBMS Version = 4.0.10-gamma-nt               Open Rowset Support = 0
Driver Name = myodbc3.dll                    ORDER BY Columns in Select
Driver ODBC Version = 03.51                  List = True
Driver Version = 03.51.05                    Outer Join Capabilities = 123
Extended Properties = DRIVER=                Outer Joins = Y
{MySQL ODBC 3.51 Driver};                    Output Parameter Availability = 4
DESC=;DATABASE=mylibraryodbc;                Pass By Ref Accessors = True
SERVER=localhost;UID=root;                   Password =
PASSWORD=saturn;PORT=;OPTION=16387;STMT=;    Persist Security Info =
File Usage = 0                               Persistent ID Type = 4
General Timeout =                            Prepare Abort Behavior = 2
GROUP BY Support = 3                         Prepare Commit Behavior = 2
Heterogeneous Table Support = 0              Procedure Term =
Identifier Case Sensitivity = 8              Prompt = 4
Initial Catalog =                            Provider Friendly Name =
Integrity Enhancement Facility               Microsoft OLE DB Provider
= False                                      for ODBC Drivers
Isolation Levels = 1118464                   Provider Name = MSDASQL.DLL
Isolation Retention = 0                      Provider Version = 02.70.7713.0
Like Escape Clause = Y                       Quoted Identifier Sensitivity = 4
Locale Identifier = 1031                     Read-Only Data Source = False
Location =                                   Reset Datasource =
Max Columns in Group By = 0                  Rowset Conversions on Command
Max Columns in Index = 32                    = True
Max Columns in Order By = 0                  Schema Term =
Max Columns in Select = 0                    Schema Usage = 0
Max Columns in Table = 0                     Server Name = localhost
Maximum Index Size = 500                     via TCP/IP
Maximum Row Size = 0                         Special Characters =
Maximum Row Size Includes                    ÇüáâåçêëèïîìÄ
BLOB = True                                  SQL Grammar Support = 1
Maximum Tables in SELECT = 31                SQL Support = 259
Mode =                                       Stored Procedures = False
Multiple Parameter Sets = True               String Functions = 491519
Multiple Results = 1                         Structured Storage = 1
Multiple Storage Objects = False             Subquery Support = 0
Multi-Table Update = False                   System Functions = 7
NULL Collation Order = 8                     Table Term = table
NULL Concatenation Behavior = 0              Time/Date Functions = 106495
Numeric Functions = 16777215                 Transaction DDL = 2
OLE DB Services = -7                         User ID =
OLE DB Version = 02.00                       User Name = root
OLE Object Support = 1                       Window Handle =

ADO Programming Techniques

Recordsets with Client-Side Cursors

Recordset objects are opened on the basis of an existing Connection object (see the previous section). In all the examples in this chapter the variable name for this object is conn.

Since ADO provides a server-side cursor as the default cursor position, a client-side cursor must be set before the Open method. The following lines demonstrate how this is done:

Dim rec As Recordset
rec.CursorLocation = adUseClient 'client-side cursor
rec.Open "SELECT  . . .    FROM  . . .    WHERE  . . .    ", _
    conn, adOpenStatic, adLockReadOnly

We note the following about the parameters of Open: In the first parameter, the SQL code is expected; in the second, the Connection variable. The third parameter describes the desired cursor type. Since for client-side cursors only the type adOpenStatic is supported, it doesn't matter what is given here. The fourth parameter specifies whether the data in the Recordset are read-only (adLockReadOnly) or whether changes are permitted (adLockOptimistic). ADO provides the locking types adLockPessimistic and adLockBatchOptimistic as well, which, however, I have not tested.

With the Open method, all the data encompassed by the SQL query are transmitted to the program. (This can take considerable time with large data sets.) You can then apply all Recordset properties and methods to run through the data (MoveNext, etc.), to search (Find), to sort locally (Sort), to change (column="new value": rec.Update), etc.

Recordset with Server-Side Cursor

You can open a Recordset with a server-side cursor with the property CursorType = adForwardOnly. You can thereby move through and read all the records of the query with MoveNext, and also change them (unless you are operating with LockType = adLockReadOnly):

Dim rec As Recordset
rec.CursorLocation = adUseServer
rec.Open "SELECT  . . .    FROM  . . .    WHERE  . . .    ", _
    conn, adOpenForwardOnly, adLockOptimistic

The only navigation method allowed is MoveNext. You cannot use Bookmarks. You cannot determine the number of records without running through all of them. (You would do better to use a separate query with SELECT COUNT(*) FROM ... .)

In spite of all these restrictions, in a few cases it can make sense to use a server-side cursor, namely, when large quantities of data are to be read sequentially. The drawbacks of the client-side cursor are thereby avoided (namely, that all the records are transmitted at once to the client).

Possible Recordset Properties

ADO provides for an almost endless number of combinations of properties of a Recordset. But the Recordset does not always contain the properties requested by Open. If a given driver does not support a particular cursor type, then ADO automatically chooses another.

As a way of testing which properties a Recordset actually possesses, you can use the program Cursortypes.vbp (see Figure 17-9). With a click of the mouse you can set the desired properties. The program then opens the Recordset, determines the actual properties, and then displays them. (Note, please, that the results of the program are not always correct for a server-side cursor. In particular, the Recordset method Supports( ... ) sometimes returns completely incorrect results.)

Click To expand
Figure 17-9: Visual Basic program to test Recordset properties

In sum, ADO + Connector/ODBC supports the following cursor combinations:

CursorLocation = Client

CursorType = Static

LockType = ReadOnly or Optimistic or BatchOptimistic

CursorLocation = Server

CursorType = ForwardOnly/Static/Dynamic

LockType = all

In the case ofCursorLocation=Server, the ODBC option Enable dynamic cursor (32) determines whether you obtain a dynamic or static cursor.

Behavior of Recordsets with Null, Dates and Times, and Suchlike

The following list describes how various MySQL data types are represented in ADO Recordsets:

  • Null: The value Null in a column of a Recordset can be determined with IsNull(rec!column).

  • Date/Time: Columns in the MySQL formats DATE, TIME, DATETIME, and TIMESTAMP are automatically transformed into the Visual Basic data type Date.Please exercise caution with TIME values: These values are supplemented with the current date. (For example, from 9:00 you may get 2003/3/17 9:00.)

  • BLOB: BLOBs are transformed into Byte arrays. Access to individual bytes is effected in the manner rec!a_blob(n),where n=0 addresses the first byte. A Byte array can be interpreted by Visual Basic as a character string. However, again caution is advised: Since internally, Visual Basic works with Unicode, a BLOB with 512 bytes corresponds in Visual Basic to a character string with 256 Unicode characters. For bytewise evaluation of the character string, the Visual Basic functions AscB, ChrB, LenB, MidB, LeftB, RightB, InStrB, etc., must be used.

  • Decimal: Columns in the MySQL format DECIMAL are metamorphosed into the Visual Basic data type Decimal, which is not to be confused with Currency. It is to be thought of more as a subtype of Variant.

  • Sets/Enums: Columns in the MySQL formats SET and ENUM are transformed into garden-variety character strings.

  • Character strings: MySQL-specific special characters in strings (\, ', ", and 0-byte) are correctly handled automatically.

Unicode Character Strings and Binary Data (AppendChunk, GetChunk)

Within Visual Basic, character strings are represented with Unicode. In storing a string, an attempt is automatically made to transform it into the character set of the MySQL server (generally latin1). If the attempt fails, then error 80040E21 is triggered. In loading a string, the characters are interpreted according to the character set of the server.

Thanks to this automatic character adaptation, the euro symbol can be stored and then loaded when the server uses the latin1 character set. In the MySQL table, the character has code 128, while in Visual Basic, the code is 8364 (AscW ).

If you wish to store Unicode character strings as such (without character adaptation), then you must use BLOB columns in the MySQL table. You can use the method AppendChunk to store a character string:

Dim rec As New Recordset
rec.CursorLocation = adUseClient
rec.Open "SELECT * FROM test LIMIT 1", conn, adOpenStatic, _
    adLockPessimistic
rec.AddNew
rec!blobcolumn.AppendChunk s
rec.Update

With GetChunk you can read your string. You have to use ActualSize to determine the size of the data:

x= rec!blobcolumn.GetChunk(rec!blobcolumn.ActualSize)

With AppendChunk and GetChunk you can process all types of binary data. (The data type of the AppendChunk parameter and GetChunk return value is Variant.) You can also transmit the data piecewise (by executing AppendChunk and GetChunk repeatedly). This is of particular utility in managing large data sets.

Remark

AppendChunk and GetChunk function only if you use a client-side cursor.

Note that Visual Basic strings are coded in Unicode Transformation Format 16 (UTF16 for short), and not UTF8, which is more prevalent under Unix. This has the consequence that ASCII characters so coded have every second byte equal to zero. Many MySQL programs have problems with such data. Visual Basic, ADO, and Connector/ODBC do not, unfortunately, offer the option of transforming strings into UTF8.

Storing Files in a Database as BLOBs and Then Reading Them

You could also use AppendChunk and Get Chunk to store files in a MySQL database and then read them. However, the Stream class from the ADO library (version 2.5 and higher) offers a more elegant alternative: The following lines show how you can open a file with LoadFromFile as a binary stream and store it in the field rec!pic (rec is a Recordset object, and pic is a BLOB column of the previously executed SELECT query):

Dim st As New Stream
st.Type = adTypeBinary
st.Open
st.LoadFromFile "C:\test1.gif"
rec!pic = st.Read

Conversely, you can read a BLOB field from the table and store it in a file with SaveToFile:

Dim st As New Stream
st.Type = adTypeBinary
st.Open
st.Write rec!pic
st.SaveToFile "C:\test2.gif", adSaveCreateOverWrite

A complete example showing the use of these methods appears later in this chapter.

Determining the AUTO_INCREMENT Number After the Insertion of a Data Record

It often occurs that after the insertion of a new data record into a table, you require the ID number of that record (that is, the value of the AUTO_INCREMENT column for the primary index). Unfortunately, this cannot be read from the Recordset with which the record was inserted. The solution to this conundrum is to determine the ID number via SELECT LAST_INSERT_ID( ). To accomplish this, you would do well to add the following function to your program:

Private Function  LastInsertedID() As Long
  Dim rec As New Recordset
  rec.CursorLocation = adUseClient
  rec.Open "SELECT LAST_INSERT_ID()", conn
  LastInsertedID = rec.Fields(0)
End Function

Bound Database Controls

Visual Basic offers several controls that can be bound directly to database queries (to Recordset objects). Within the control, the contents of the Recordset (or perhaps only of the currently active data record) are displayed. Such controls offer a great savings in effort in the representation of database queries. Some controls even offer the possibility of altering data directly. In the ideal case, one can program a database interface with a minimal amount of code (see Figure 17-10).

Click To expand
Figure 17-10: Visual Basic example program with the MSHFlexGrid for representing data from the mylibrary database

In reality, things do not always go so smoothly. In general, you must take care in your work with Connector/ODBC that for bound controls, too, you work with a client-side cursor. Furthermore, the alteration of data does not always function reliably, so that (except in well-tested special cases) one is advised not to use this feature.

Apart from that, the use of bound controls is independent of whether you use a Microsoft database system or MySQL as the data source. Additional information on using bound controls can be found in any book on database programming with Visual Basic.

Executing SQL Commands Directly

ADO Recordsets can simplify your dealings with query results, but it is often necessary to execute SQL commands directly. To do this, use the Execute method of the Connection object:

conn.Execute "INSERT INTO table (a, b) VALUES ('x', 'y')"

As with other programming languages, with Visual Basic you also run into the problem that you have to format various data in conformity with the rules of MySQL. In the following, you will find some auxiliary functions to help you in this task:

  • Floating-Point Numbers: With floating-point numbers you must take into account that Visual Basic normally formats such numbers according to the country for which the language has been customized (for example, in Germany one has 3,14159265 instead of 3.14159265). If your program is to function equally well everywhere in the known universe, you should transform your floating-point numbers into character strings with the Visual Basic function Str.

  • Date: The following Format instruction formats the date contained in x in accord with MySQL conventions:

    Format(x, "'yyyy-mm-dd Hh:Nn:Ss'")

  • Character Strings: Quote places a backslash character before the backslash and before the single- and double-quote characters, and it replaces the byte code 0 with \0:

    Function  Quote(tmp)
  tmp = Replace(tmp, "\", "\\")
  tmp = Replace(tmp, """", "\""")
  tmp = Replace(tmp, "'", "\'")
  Quote = Replace(tmp, Chr(0), "\0")
End Function

  • Binary Data: HexCode transforms a byte array of arbitrary length into a hexadecimal character string of the form 0x0102031232:

    Function  HexCode$(bytedata() As Byte)
  Dim i&
  Dim tmp$
  tmp = ""
  For i = LBound(bytedata) To UBound(bytedata)
      If bytedata(i) <= 15 Then
  tmp = tmp + "0" + Hex(bytedata(i))
      Else
  tmp = tmp + Hex(bytedata(i))
      End If
  Next
  HexCode = "0x" + tmp
End Function

You might use this function in the following way:

conn.Execute "INSERT INTO table (col) VALUES (" + HexCode( ... )+ ")"

HexCodeStr functions in principle like Hexcode, except that it expects data bytewise in a character string. Therefore, you can also store Unicode character strings with HexCodeStr:

s =  "Unicode character string  . . .    "
conn.Execute "INSERT INTO table (col) VALUES (" + HexCode(s) + ")"

Note, however, that Visual Basic character strings are UTF16 encoded, and not in the form UTF8, which is more usual under Unix:

Function  HexCodeStr$(bytedata$)
  Dim i&, b&
  Dim tmp$
  tmp = ""
  For i = 1 To  LenB(bytedata)
    b =  AscB(MidB(bytedata, i, 1))
    If b <= 15 Then
  tmp = tmp + "0" + Hex(b)
    Else
  tmp = tmp + Hex(b)
    End If
  Next
  HexCodeStr = "0x" + tmp
End Function

Example: authors Column for the titles Table

In Chapter 5 we described a possible extension for the titles table of the mylibrary database, whereby the table would be equipped with an additional authors column in which the authors' names would be stored. To be sure, this is redundant, but it will speed up read access to the database.

Click To expand
Figure 17-11: Visual Basic example program for changing the titles table

The example program presented here (see Figure 17-11) shows how the authors column can be filled with data. For this, we simply run through all the titles records, determine the authors for each book, combine the authors into a character string, and store them.

Remark

Our program assumes that the titles table has been enlarged to include an (as of yet empty) authors column of type VARCHAR(255):

ALTER TABLE titles ADD authors VARCHAR(255)

Variant 1: Changing Data with Read/Write Recordsets

All of the code is contained in the event procedure Command2a_Click. This procedure assumes that the global variable conn creates the connection to the database (Connection object).

In the program, two Recordsets are opened. One of them, titles, refers directly to the like-named table and enables changes to it (LockingType=adLockOptimistic). The second, authors, contains a list of all titleID values and their authors. The Recordset is sorted locally after being read in (property Sort), and apart from the database, in order not to put unnecessary demands on resources.

The following loop runs through all the titleID values of the titles Recordset. With Find, the associated authors in the authors Recordset are determined and joined into the character string authors_str.This character string is then stored:

'  example vb6 \authors_for_titles\form1.frm
Private Sub Command2a_Click()
  Dim authors_str As String
  Dim titles As New Recordset
  Dim authors As New Recordset
  '  titles Recordset: read/write
  titles.CursorLocation = adUseClient
  titles.Open "SELECT titleID, authors FROM titles", _
    conn, adOpenStatic, adLockOptimistic
  '  authors Recordset: readonly, disconnected
  authors.CursorLocation = adUseClient
  authors.Open "SELECT titleID, authname " _
              "FROM authors, rel_title_author " & _
              "WHERE authors.authID=rel_title_author.authID", _
              conn, adOpenStatic, adLockReadOnly
  authors.Sort = "titleID, authname"
  Set authors.ActiveConnection = Nothing
  '  loop over all titles
  While Not titles.EOF
    authors_str = ""
    '  loop over all authors of this title
    authors.MoveFirst
    authors.Find "titleID=" & titles!titleID
    While Not authors.EOF
      If authors_str <> "" Then authors_str = authors_str + "; "
      authors_str = authors_str & authors!authName
      authors.MoveNext
      authors.Find "titleID=" & titles!titleID
    Wend
    '  store author list
    If authors_str <> "" Then
      titles!authors = authors_str
      titles.Update
    End If
    titles.MoveNext
  Wend
End Sub

Variant 2: Changing Data with conn.Execute

In our second variant, the authors Recordset is opened just as in variant 1. The structure of the loop is a bit different in how it finds all the authors of the current titleID value. The authors are stored temporarily in the variable authors_str.

What is new in comparison to variant 1 is the instruction conn.Execute, with which the list of authors is stored in the titles table with the help of an UPDATE command. The function Quote is necessary to ensure that the SQL command is correct even when the author name has special characters such as ':

Private Sub  Command2b_Click()
  Dim authors_str, titleID
  Dim authors As New Recordset
  '  authors Recordset: client-side, readonly, disconnected
  authors.CursorLocation = adUseClient
  authors.Open "SELECT titleID, authname " & _
      "FROM authors, rel_title_author " & _
      "WHERE authors.authID=rel_title_author.authID " & _
      "ORDER BY titleID, authName", _
      conn, adOpenStatic, adLockReadOnly
  authors.Sort = "titleID, authname"
  Set authors.ActiveConnection = Nothing
  '  loop over all titles (titleID)
  While Not authors.EOF
    titleID = authors!titleID
    authors_str = ""
    '  search for all authors of the current titleID value
    authors_str = ""
    Do While Not authors.EOF
  ' at  the next titleID jump out of the inner loop
  If authors!titleID <> titleID Then
            Exit Do
  End If
  '  add the author name to authors_str
  If Not IsNull(authors!authName) Then
    If authors_str <> "" Then authors_str = authors_str + "; "
    authors_str = authors_str & authors!authName
  End If
  authors.MoveNext
    Loop
    '  store author list
    If authors_str <> "" Then
  conn.Execute "UPDATE titles " & _
      "SET authors = '" & Quote(authors_str) & "'" & _
      "WHERE titleID=" & titleID
    End If
  Wend
End Sub
'  place \ before ', ", and \ ,  replace Chr(0) by \0
Private Function  Quote(tmp)
  tmp = Replace(tmp, "\", "\\")
  tmp = Replace(tmp, """", "\""")
  tmp = Replace(tmp, "'", "\'")
  Quote = Replace(tmp, Chr(0), "\0")
End Function

Example: Adding a New Book Title

With the following example program you can store a new book title in the mylibrary database. The book title consists of the title itself, one or more authors, and a publisher (see Figure 17-12). The program does not test whether the authors or publisher already exists in the database. The goal of this program is simply to demonstrate the principal methods of inserting data into linked tables.

Click To expand
Figure 17-12: Visual Basic example program for inserting a new book title

Variant 1: Changing Data with Read/Write Recordsets

At the beginning of the procedure, four Recordset objects are opened, with whose help the data are to be stored. Without LIMIT 1, the SELECT statement would retrieve all matching Recordsets. But we are not interested in these (we want only to change data here), so this would be a waste of time. What we want is an ADO object pointing to this table, and to get one, we have to execute a SELECT statement. The use of LIMIT 1 thus speeds things up for us. (If you would like to improve the efficiency of the code, you should define the four Recordset variables globally and open them in Form_Load. You thereby avoid the Recordsets having to be opened for each storage event.)

The commands for storing the data from the three text fields txtPublisher, txtTitle, and txtAuthors are easy to understand. Worthy of note is the use of the auxiliary function LastInsertedID( ), described above, for determining the most recently added AUTO_INCREMENT value:

'  example vb6 \insert_new_title\form1.frm
Private Sub  SaveData_WithRecordsets()
  Dim i&, titleID&, authID&, publID&
  Dim authors_array
  Dim authors As New Recordset, titles As New Recordset
  Dim publishers As New Recordset, rel_title_author As New Recordset
  '  open Recordsets
  authors.CursorLocation = adUseClient
  titles.CursorLocation = adUseClient
  publishers.CursorLocation = adUseClient
  rel_title_author.CursorLocation = adUseClient
  authors.Open "SELECT * FROM authors LIMIT 1", _
    conn, adOpenStatic, adLockOptimistic
  titles.Open "SELECT * FROM titles LIMIT 1", _
    conn, adOpenStatic, adLockOptimistic
  publishers.Open "SELECT * FROM publishers LIMIT 1", _
    conn, adOpenStatic, adLockOptimistic
  rel_title_author.Open "SELECT * FROM rel_title_author LIMIT 1", _
    conn, adOpenStatic, adLockOptimistic
  '  store publisher (if given)
  If Trim(txtPublisher) <> "" Then
    publishers.AddNew
    publishers!publName = Trim(txtPublisher)
    publishers.Update
    publID = LastInsertedID()
  End If
      'store book title (with publID reference as needed)
  titles.AddNew
  titles!Title = Trim(txtTitle)
  If publID <> 0 Then titles!publID = publID
  titles.Update
  titleID = LastInsertedID()
      'store authors and make entries in rel_title_author
  authors_array = Split(txtAuthor, ";")
  For i = LBound(authors_array) To UBound(authors_array)
    authors.AddNew
    authors!authName = Trim(authors_array(i))
    authors.Update
    authID = LastInsertedID()
    rel_title_author.AddNew
    rel_title_author!titleID = titleID
    rel_title_author!authID = authID
    rel_title_author.Update
  Next
End Sub
Private Function LastInsertedID()  . . .    see the heading "ADO Programming Techniques"

Variant 2: Changing Data with conn.Execute

In our second variant, the code is more like that to which you are accustomed in MySQL programming with languages such as PHP or Perl: You need to cobble together the SQL commands painstakingly, taking care to treat special characters correctly. (For this, the auxiliary function Quote will be used.) The structure of the code is otherwise like that of the first variant:

Private Sub SaveData_WithSQLCommands()
  Dim i&, titleID&, authID&, publID&
  Dim authors_array
  '  store publisher (if given)
  If Trim(txtPublisher) <> "" Then
    conn.Execute "INSERT INTO publishers (publName) " & _
      "VALUES ('" & Quote(Trim(txtPublisher)) & "')"
    publID = LastInsertedID()
  End If
  '  store book title (with publID as needed)
  conn.Execute "INSERT INTO titles (title, publID) " & _
      "VALUES ('" & Quote(Trim(txtTitle)) & "', " & _
      IIf(publID <> 0, publID, "NULL") & ")"
  titleID = LastInsertedID()
  '  store authors and add entries to rel_title_author
  authors_array = Split(txtAuthor, ";")
  For i = LBound(authors_array) To UBound(authors_array)
    conn.Execute "INSERT INTO authors (authName) " & _
      "VALUES ('" & Quote(Trim(authors_array(i))) & "')"
    authID = LastInsertedID()
    conn.Execute "INSERT INTO rel_title_author " & _
      "  (titleID, authID) " & _
      "VALUES (" & titleID & ", " &  authID & ")"
  Next
End Sub
Private Function  Quote(tmp)  . . .    see heading "ADO Programming Techniques"

Example: Storing an Image File in BLOB Format and then Reading It

The following example program reads the file test.jpg from the current directory and stores it in a table (Command2_Click). In a second step, the binary data are read from the table and stored in the file test1.jpg (Command3_Click). As a visual check whether everything worked as it should have, the new file will be displayed in a Picture control (see Figure 17-13). For storing the image, at the beginning of the program a table is created in the following way and then deleted at the end of the program:

CREATE TABLE IF NOT EXISTS testpic
    (id INT NOT NULL AUTO_INCREMENT, pic BLOB, PRIMARY KEY (id))
Click To expand
Figure 17-13: The photograph is loaded from a BLOB field and displayed

Here is the code:

'Example vb6\blob\form1.frm
Option Explicit
Dim conn As Connection
Dim id As Long
    ' create connection to MySQL
    Private Sub Command1_Click()
  Set conn = New Connection
  'Options: 16395 = 16384 + 8 + 2 + 1
  '    Don't Optimize Column Width +
  '    Allow Big Results + Change BIGINT Columns to INT
  conn.ConnectionString = "Provider=MSDASQL;" + _
    "DRIVER={MySQL ODBC 3.51 Driver};" + _
    "Server=localhost;UID=root;PWD=saturn;" + _
    "database=mylibraryodbc;Option=16395"
  conn.Open
  '  create table
  conn.Execute "CREATE TABLE IF NOT EXISTS testpic " + _
  "(id INT NOT NULL AUTO_INCREMENT, pic BLOB, PRIMARY KEY (id))"
  Command2.Enabled = True
End Sub
'  read file into a Stream object and
    'store it in the table testpic
    Private Sub Command2_Click()
  Dim rec As New Recordset
  Dim fname As String
  Dim st As New Stream
  '  open file
  fname = App.Path + "\test.jpg"
  st.Type = adTypeBinary
  st.Open
  st.LoadFromFile fname
          'store the file in the table
  rec.CursorLocation = adUseClient
  rec.Open "SELECT * FROM testpic LIMIT 1", conn, adOpenKeyset, adLockOptimistic
  rec.AddNew
  rec!pic = st.Read
  rec.Update
  rec.Close
  st.Close
  '  note id (AUTO_INCREMENT number)
  id = LastInsertedID()
  Command3.Enabled = True
End Sub
'  read the BLOB from the table testpic and store it in a new file
Private Sub Command3_Click()
  Dim rec As New Recordset
  Dim fname As String
  Dim st As New Stream
  '  Stream object for the new file
  fname = App.Path + "\test1.jpg"
  st.Type = adTypeBinary
  st.Open
  '  read the BLOB into the Stream object
  rec.CursorLocation = adUseClient
  rec.Open "SELECT pic FROM testpic WHERE id = " & id,  conn, _
      adOpenKeyset, adLockReadOnly
  st.Write rec!pic
  rec.Close
  'store the Stream in the file
  st.SaveToFile fname, adSaveCreateOverWrite
  st.Close
  '  display the file in a Picture control
  Picture1.Picture = LoadPicture(fname)
End Sub


MyVbQL (Visual Basic Programming Without ADO/ODBC)

The ADO programming based on Connector/ODBC introduced in the previous section works, but it is not particularly efficient. This section offers as an alternative the object library MyVbQL. This library, developed in Visual Basic, offers Visual Basic programmers more direct access to MySQL. To this end, the MyVbQL objects MYSQL_CONNECTION, MYSQL_RS, MYSQL_FIELD, and MYSQL_ERR display functionality similar to that of the ADO objects Connection, Recordset, Field, and Error, without, however, being compatible with them.

MyVbQL is available without charge. The GNU Library General Public License is used. This means that libraries may also be used in commercial applications (see also http://www.gnu.org/copyleft/library.html).

For this book, MyVbQL 2 was tested.

The DLL and underlying Visual Basic code can be found on the Internet:

http://www.icarz.com/mysql/

MyVbQL builds on the MySQL client library libmySQL.dll. Access to MySQL via MyVbQL takes place according to the following scheme: Visual Basic ® MyVbQL Object library ® libmySQL.dll ® MySQL.

In comparison to ADO + Connector/ODBC, MyVbQL code is generally faster. A further considerable advantage is that MyVbQL is much less complex than ADO + Connector/ODBC. In other words, there are fewer functions, and they are generally less likely to cause errors. It is also quite practical that in distributing MyVbQL programs, one does not have to include ADO libraries, which inflate the setup files considerably. Unfortunately, there are some drawbacks as well:

  • The time for establishing the connection is somewhat longer (particularly if MySQL is running on another computer under another operating system).

  • MyVbQL currently does not support BLOBs.

  • MyVbQL is not ADO-compatible. It therefore takes some effort to convert existing ADO code to MyVbQL.

  • The MYSQL_RS object cannot be used as the data basis for bound database controls, which makes the use of ADO database controls in combination with MyVbQL impossible.

  • The included documentation for MyVbQL is in the nature of a simple reference. The use of MyVbQL therefore assumes some background in Visual Basic and databases.

  • MyVbQL will presumably not be developed further. (The tested version 2 dates from January 2001 and has not been changed since then.)

Installation

In order for MyVbQL to be able to be used, the two DLLs MyVbQL.dll and libmysql.dll must be installed on the computer. Both files are located in MyVbQL.zip on the web site http://www.icarz.com/mysql/.

With MyVbQL.dll we are dealing with an ActiveX DLL. This file may be copied into the directory of your choice (e.g., the Windows system directory). Furthermore, the file must be registered in the Windows registration database. This takes place automatically as soon as the library is transformed for the first time via PROJECT |REFERENCES into a Visual Basic project.

The file libmysql.dll is a traditional DLL. During program development, it must be located in the directory of the Visual Basic development environment (e.g., C:\Programs\Visual Basic 6). Once your Visual Basic program is complete and compiled, then libmysql.dll must be copied into the directory of the compiled Visual Basic program.

Caution

Note that with every new MySQL installation, a (more current) version of libmysql.dll is installed, usually in the directory C:\mysql\bin.

It is difficult to say whether MyVbQL should be used with the included (old) version of libmysql.dll, or with themorecurrent onefrom the MySQL installation. On the one hand, in the current version, various errors have been corrected that still exist in the version provided with MyVbQL. On theother hand, the new version could have some incompatibilities.

In no case should you copy the version from MyVbQL into the Windows system directory (or to another place where the file might be used by programs other than MyVbQL). The reason is that in addition to MyVbQL, thereare other programs and libraries that depend on libmysql.dll,andthese programs generally expect the most current version.

Application

In order to use MyVbQL in your own Visual Basic project you must simply create a reference to the library. As soon as the library is registered, you will find it under the name MySQL Visual Basic API in the dialog PROJECT |REFERENCES. If the library has not yet been registered, then use the button BROWSE in order to load the DLL. You can then use the four MYSQL_xxx objects as you would normal Visual Basic objects.

Example

Again we shall use as an example the program that stores a character string with all the authors belonging to a given book title in the authors column of the titles table.

For establishing the connection you create an object of type MYSQL_CONNECTION and pass the computer name, user name, password, and database name to the method OpenConnection:

'  authors_for_titles_myvbql\form1.frm
'  establish connection to MySQL
Dim conn As New MYSQL_CONNECTION
Private Sub Command1_Click()
  conn.OpenConnection "localhost", "root", "xxx", "mylibraryodbc"
  Command2.Enabled = True
End Sub

MYSQL_RS objects are derived with the method Execute from the MYSQL_CONNECTION object. Access to individual fields of the current record is effected with rs.Fields("columnName").Value (instead of rs!columnName, as with ADO).

To alter data in MYSQL_RS objects, simply assign a new value to the Value property of the affected column and store the changes with the Update method. (You can also insert a new record with AddNew or delete one with Delete, but these are not demonstrated in our example.)

Caution

Be sure to close every MYSQL_RS object explicitly with Close after you are through with it. If you forget this, then the piece of memory for storing the object will never be released. (This is due to a known, but not fixed, error in the MyVbQL library.)

'  change titles table
Private Sub Command2_Click()
  Dim authors_str, titleID
  Dim titles As MYSQL_RS
  Dim authors As MYSQL_RS
  '  titles Recordset for a loop over all titles
  '  and for altering the authors column
  Set titles = conn.Execute( _
      "SELECT titleID, authors FROM titles")
  '  loop over all book titles
  While Not titles.EOF
    titleID = titles.Fields("titleID").Value
    '  authors Recordset to read the authors of a title
    Set authors = conn.Execute( _
  "SELECT authname FROM authors, rel_title_author " & _
  "WHERE authors.authID=rel_title_author.authID " & _
  "    AND rel_title_author.titleID = " &  titleID & " " & _
  "ORDER BY authName")
    '  loop over all authors
    authors_str = ""
    Do While Not authors.EOF
  '  assemble string with author names
  If Not IsNull(authors.Fields("authName").Value) Then
    If authors_str <> "" Then authors_str = authors_str + "; "
    authors_str = authors_str & authors.Fields("authName").Value
  End If
  authors.MoveNext
    Loop
    authors.CloseRecordset
    '  save authors field in titles Recordset object
    If Len(authors_str) > 254 Then _
  authors_str = Left(authors_str, 254)
    titles.Fields("authors").Value = authors_str
    titles.Update
    titles.MoveNext
  Wend
  titles.CloseRecordset
End Sub


ADO.NET (Visual Basic .NET, C#)

The ADO library is also available for the programming languages Visual Basic .NET and C#. You can therefore create database applications like those that we have introduced. However, ADO in combination with VB.NET or C# is not very popular, for several reasons:

  • In VB.NET/C#, the library ADO.NET is available, which offers numerous advantages in comparison to ADO. The abbreviation ADO.NET stands for ActiveX Data Objects .NET, but internally, ADO.NET has nothing to do with ADO. It is rather a .NET-conforming new implementation in the name space System.Data.

  • ADO can be executed in VB.NET/C# only over the COM compatibility layer. This is considered inefficient (although the speed differences with most database applications are scarcely noticeable any longer). Furthermore, many advantages of pure .NET applications are lost (the .NET security system, the relatively simple transfer without DLL version problems, etc.).

In short, those who use VB.NET/C# generally also use ADO.NET. This section gives an overview of the available variants and discusses some of the basic concepts.

Note, please, that the classes and methods of ADO.NET are completely incompatible with those of ADO. Microsoft has not simply made an existing library .NET compatible, but has completely rethought its structure, and has realized a host of new concepts and approaches.

Remark

The examples presented in this chapter use, with few exceptions, VB.NET as the programming language. Friends of C# will find equivalent C# projects to most of these examples in the source code (www.apress.com).

All examples were developed with the VS.NET development environment; that is, we have used the usual default settings for libraries and imports.
Remark

It is basically true that MySQL programming with ADO.NET is not particularly MySQL-specific. Once the connection has been made, the rest is like any other ADO.NET program. The only major difference is that instead of the classes OleDbXxx and SqlXxx described in all the ADO.NET books, the largely equivalent classes OdbcXxx and MySQLXxx must beused.

Due tospace limitations, it is not possible in this book to give an introduction to ADO.NET. I therefore must assume that you are familiar with the basics of ADO.NET. If that is not the case, then before reading this section, you should get yourself a book on ADO.NET.

Communication Between ADO.NET and MySQL

Many roads lead from ADO.NET to MySQL, regardless of whether you prefer to work with C# or VB.NET. The following list gives an overview:

  • Connector/ODBC + ODBC data provider: In this variant, communication takes place via ODBC. The interface between MySQL and ODBC is provided by Connector/ODBC. As data provider for ADO.NET, the ODBC data provider (name space Microsoft.Data.Odbd) is used. This data provider is not provided with .NET 1.0 and must be installed separately. The ODBC provider will be an integral component of ADO.NET beginning with .NET 1.1:

    VB.NET/C# ® ADO.NET ® ODBC data provider ® Connector/ODBC ®MySQL

  • Connector/ODBC + OLEDB data provider: In this variant as well, communication takes place via ODBC. However, the access route to MySQL is now extended by an additional component, the OLEDB provider for ODBC: VB.NET/C# ® ADO.NET ® OLEDB data provider ® OLEDB provider for ODBC ®Connector/ODBC ® MySQL

  • Managed data provider for ADO.NET: For efficient communication between ADO.NET and MySQL, it is best to use a special provider for ADO.NET: VB.NET/C# ® ADO.NET ® MySQL data provider ® MySQL

The obvious advantage of such a provider is that the communication path is the shortest. This represents fewer compatibility problems between the individual layers and greater efficiency. Moreover, specially tailored providers for MySQL can provide MySQL-specific functions (such as support of the compressed communication protocol). These additional functions can be useful, but they have the drawback of making difficult a later change of provider (or even the database system).

Fortnately, there is now a host of such providers. In February 2003 I found the following drivers on the Internet:

  • ByteFX.Data, alias MySQLNet (Open Source, LGPL): This open-source driver was still in beta test stage in February 2003, but it made a very good impression. Unfortunately, there is no documentation at all. The driver also contains classes for programming of database applications with PostgresSQL:

    http://sourceforge.net/projects/mysqlnet/

  • MySQLDriverCS (Open Source, GPL): This open-source driver is, according to its developers, fully mature. However, the driver currently provides only a fraction of the usual .NET driver classes. (For example, MySQLDataAdapter, MySQLParameter,and MySQLCommandBuilder are lacking.)

    http://sourceforge.net/projects/mysqldrivercs/

  • dbProvider from eInfoDesign: According to its creators, this commercial driver completely implements the .NET data interface. The reduced-functionality Personal Edition (no support for transactions, only latin1 allowed as character set) is available free of charge:

    http://www.einfodesigns.com/dbProvider_info.aspx

  • MySQLDirect .NET from CoreLab: This is yet another commercial driver. It is based on the MySQL client library libmysql.dll (which is included and can be installed directly into the directory Windows\Systems32). There is a free test version available. A particular advantage of this driver is its extensive documentation (*.chm help file).

It was, of course, impossible to test all the variants introduced here. I have tried out connections with the combination Connector/ODBC + ODBC provider as well as with the providers ByteFX, dbProvider, and MySQLDirect .NET. However, all further tests were made exclusively with Connector/ODBC + ODBC provider.

The ODBC example should function with the three MySQL providers without noteworthy changes if you change the lines for establishing the connection and replace OdbcXxx by MySQLXxx or MySqlXxx. (The use of upper- and lowercase in the name MySQL differs among the three providers. In my experiments it was sometimes necessary to use cast operations. There are also some small syntactic variants in the initialization of objects.)

Establishing the Connection with the ODBC Data Provider

Assumptions

Before you start programming, three conditions must be satisfied:

  • ODBC data provider: The library Microsoft.Data.Odbc has been included with the .NET Framework only since version 1.1. If you are working with version 1.0, you must download the *.msi file from the address below. A double click installs the provider in the global assembly cache of the .NET Framework, making it available to all .NET programs:

    http://www.microsoft.com/downloads/release.asp?ReleaseID=35715

    A reference to this library must now be established for each project. To save typing, one should include the instruction Imports Microsoft.Data.Odbc or using Microsoft.Data.Odbc at the beginning of the code.

  • Connector/ODBC: The second requirement for access to MySQL is Connector/ODBC, whose installation we have described in Chapter 2.

  • MDAC update: The third condition is to install an update (bug fix) to the Microsoft database components (MDAC 2.7). An error in the ODBC driver manager leads to the exception SQL_NO_DATA being triggered in ODBC.NET programs when an empty string is read via ODBC. The update can be found at

    http://support.microsoft.com/default.aspx?scid=kb;EN-US;q319243

Establishing the Connection

The connection looks very much as it did with ADO. The main difference is that now an OdbcConnection object is used, and the connection string can be passed to the constructor. The connection string is exactly the same as with ADO. As a rule, the Options setting should be 3. (This corresponds to the options Don't optimize column width (1) and Return matching rows (2). The option Change BIGINT into INT (16384) is not required with ADO.NET.)

'  VB.NET
Dim odbcconn As New Microsoft.Data.Odbc.OdbcConnection( _
  "Driver={MySQL ODBC 3.51 Driver};Server=localhost;" + _
  "Database=mylibraryodbc;UID=root;PWD=saturn;Options=3")
odbcconn.Open()
// C#
OdbcConnection odbcconn = new OdbcConnection(
  "Driver={MySQL ODBC 3.51 Driver};Server=localhost;" +
  "Database=mylibraryodbc;UID=root;PWD=saturn;Options=3");
odbcconn.Open();

VB.NET Example

The following small program displays an alphabetic list of all publishers in mylibraryodbc.publishers in a console window. After the connection is established with OdbcCommand, a SELECT query is executed. The result is evaluated with an OdbdDataReader object in a loop:

'  example vbnet/odbc_connect/module1.vb
Option Strict On
Imports System.Data
Imports Microsoft.Data.Odbc
Module Module1
  Dim odbcconn As OdbcConnection
  Sub Main()
    odbcconn = New OdbcConnection( _
  "Driver=MySQL ODBC 3.51 Driver;Server=localhost;" + _
  "Database=mylibraryodbc;UID=root;PWD=saturn;Options=16387")
    odbcconn.Open()
    read_publishers_datareader()
    odbcconn.Close()
  End Sub
  '  example for OdbcDataReader
  Sub read_publishers_datareader()
    Dim com As OdbcCommand
    Dim dr As OdbcDataReader
    com = New OdbcCommand( _
  "SELECT publID, publName FROM publishers ORDER BY publName", _
  odbcconn)
    dr = com.ExecuteReader()
    While dr.Read()
  Console.WriteLine("id: {0} name: {1}", dr!publID, dr!publName)
    End While
    dr.Close()
  End Sub
End Module

C# Example

The equivalent C# code looks like this:

// example csharp\odbc_connect\Class1.cs
using System;
using Microsoft.Data.Odbc;
namespace odbc_connect  {
      class mainclass{
    [STAThread]
    static void  Main(string[] args) {
  Odbctest tst = new Odbctest();
  tst.ReadPublishersDataset();
  Console.WriteLine("Return drŸcken");
  Console.ReadLine();
      }
  }
  class  Odbctest  {
    OdbcConnection odbcconn;
    public Odbctest() {         // use constructor to connect
  odbcconn = new OdbcConnection(
    "Driver={MySQL ODBC 3.51 Driver};Server=localhost;" +
    "Database=mylibraryodbc;UID=root;PWD=saturn;Options=16387");
  odbcconn.Open();
    }
    public void ReadPublishersDataset() {
  OdbcCommand com = new OdbcCommand(
    "SELECT publID, publName FROM publishers ORDER BY publName",
    odbcconn);
  OdbcDataReader dr = com.ExecuteReader();
  while(dr.Read()) {
    Console.WriteLine("id: {0} name: {1}",
    dr["publID"], dr["publName"]);
  }
  dr.Close();
    }
  }
}

Establishing the Connection with the ByteFX.Data Provider

Assumptions

The ByteFX.Data provider is available at http://sourceforge.net/projects/mysqlnet/ as a simple ZIP file. It contains two files: ByteFX.Data.dll and SharpZipLib.dll. Both files must be copied either into the global assembly cache (GAC) or into the project directory. Then you must place a reference to both files in your new project (ByteFX.Data contains the actual ADO classes, while SharpZipLib.dll has the compression functions).

To avoid having to specify ByteFX.Data.MySQLClient.xxx repeatedly, you should place an Imports or using instruction at the beginning of your code:

Imports ByteFX.Data.MySQLClient   '   VB.NET
using ByteFX.Data.MySQLClient; // C#

The information in this section is based on the tested version 0.65.

Establishing the Connection

The connection is now made with a MySQLConnection object. The string with connection data is made up of the following pieces:

Data Source | host name or IP address of the MySQL server
Initial Catalog | name of the database
User ID | user name
Password | password
Use Compression | compress communication between client and server? (True/False)
Dim myconn As New MySQLConnection( _
  "Data Source=localhost;Initial Catalog=mylibraryodbc;" + _
  "User ID=root;PWD=saturn;Use Compression=true")
myconn.Open()

In contrast to the other providers that we have discussed, ByteFX supports compressed data exchange to the server (connection parameter Use Compression). Whether compression is being used can be determined from the property UseCompression.

The source code for this book includes examples in VB.NET and C#. After the connection is established, an alphabetic list of all publishers is displayed in a console window.

Establishing the Connection with dbProvider (eInfoDesign)

Assumptions

The freely available Personal Edition of dbProvider from eInfoDesign is available at http://www.einfodesigns.com/dbProvider_info.aspx as a *.msi file. You must register before downloading the file.

Installation of the file dbProvider.dll is by default into the directory Programs\eInfoDesigns\dbProvider Personal Edition. You should copy the *.dll file either into the global assembly cache or into your project directory. Then you must set up a reference in your new project to the *.dll file.

To avoid having to specify the name space of the dbProvider classes repeatedly in your code, you should place an Imports or using instruction at the beginning of your code:

Imports eInfoDesigns.dbProvider.MySqlClient   '   VB.NET
using eInfoDesigns.dbProvider.MySqlClient; // C#

The information in this section is based on the Personal Edition, version 1.6.977 (version number according to DLL properties).

Establishing the Connection

The character string with the connection data for the MySQLConnection object consists of the following pieces:

Data Source | host name or IP address of the MySQL server
Database | name of the database
User ID | user name
Password | password
COMMAND LOGGING | command logging? (true/false)
POOLING | use connection pooling? (true/false)
myconn = New MySqlConnection( _
  "Data Source=localhost;Database=mylibraryodbc;" + _
  "User ID=root;Password=saturn")
myconn.Open()

The source code for this book includes examples in VB.NET and C#. After the connection is established, an alphabetic list of all publishers is displayed in a console window.

Establishing the Connection with MySQLDirect .NET (CoreLab)

Assumptions

The test version of MySQLDirect .NET can be downloaded from the site http://crlab.com/download.html. The installation program (*.msi) installs two *.dll files into the global assembly cache. Furthermore, after asking, it copies libmysql.dll into the Windows system directory. (If you have a newer version of this client library from a MySQL installation, you should use it. Copy the file if necessary from C:\mysql\bin into the Windows system directory.)

Add a reference in your new project to the library CoreLab.MySQL. To save typing, add the following Imports/using instruction:

Imports CoreLab.MySql ' VB.NET
using CoreLab.MySql; // C#

MySQLDirect .NET was the only one of the providers tested that had acceptable on-line documentation. (However, help must be invoked with START |PROGRAMS |CORELAB. The shortcut F1 does not work in the develoment environment.) The version tested was 1.21.6.

Establishing the Connection

The string with connection data for the MySqlConnection object can be assembled from the following parts:

Data Source or Host or Server | host name or IP Address of the MySQL server
Database or Initial Catalog | name of the database
User or User ID | user name
Password or PWD | password
Port | IP port (default 3306)
Connection Timeout | maximum wait for connection (default 15 seconds)
Dim myconn As New MySqlConnection( _
  "Host=localhost;Database=mylibraryodbc;" + _
  "User=root;Password=saturn;Connection Timeout=30")
myconn.Open()

For the MySqlDirect .NET provider as well there is a C# example in the source code for this book.

Programming Techniques

Remark

As we have already mentioned, the remaining sections relate exclusively to Connector/ODBC in combination with the ODBC data provider (though we may occasionally make reference to other providers).

If you are not keen on using ODBC, but rather one of the other providers that we have described, you must establish a reference to the appropriate library, change the Imports/using instruction, replace OdbcXxx by MySqlXxx, and adapt the connection string according to the provider's required syntax. Depending on the provider, you may also have to make minor changes in the code (e.g., cast operators, parameter syntax in SQL commands).

All the examples assume that odbcconn is an open OdbcConnection object that refers to a suitable MySQL database.

Executing SQL Commands (OdbcCommand)

To execute SQL commands, you need an OdbcCommand object. The basic OdbcConnection object and the SQL command can be passed to the constructor or handled separately. (The latter is especially useful if you wish to execute several SQL commands with a single OdbcCommand object.)

There are three ways to execute SQL commands:

  • ExecuteNonQuery is used for all commands that do not immediately return a result (e.g., UPDATE, INSERT, DELETE):

    Dim com As New OdbcCommand()
com.Connection = odbcconn
com.CommandText = "INSERT INTO table (colname) VALUES ('abc')"
com.ExecuteNonQuery()

  • ExecuteReader is used if you wish to execute a SELECT query and read the result with OdbcDataReader (see the example somewhat later in this chapter).

  • ExecuteScalar is used when the command returns a single value (e.g., SELECT COUNT(*) FROM table); an example follows shortly.

If an error occurs during the execution of an SQL command, an OdbcException occurs, which can be caught with Try/Catch:

Try
  com.ExecuteNonQuery()
Catch e As OdbcException
  MsgBox(e.Message)
  Stop
End Try

In formulating an SQL command, you must heed the usual MySQL peculiarities: In character strings, the characters ', ", \, and the zero-byte must be indicated by \', \", \\, and \0. Dates must also be correctly formatted (2003-12-31 23:59:59). You can save effort with INSERT and DELETE commands by using commands with parameters, which we now discuss.

Commands with Parameters (ODBC Parameters)

Parameters in SQL commands are indicated by ?. Then, for each parameter, you use Parameters.Add to specify the name and data type. Make sure that you present the parameters in the correct order:

Dim com As New OdbcCommand()
com.Connection = odbcconn
com.CommandText = _
  "INSERT INTO tablename (a_date, a_text, a_blob) VALUES(?, ?, ?)"
com.Parameters.Add("pdate", OdbcType.DateTime)
com.Parameters.Add("ptext", OdbcType.Text)
com.Parameters.Add("pblob", OdbcType.Binary)

Whenever you wish to execute an INSERT command, you must now pass the appropriate data to the parameters, which could look like this:

Dim bin(49) As Byte   '50 Bytes
   . . .
com.Parameters!pdate.Value = Now
com.Parameters!ptext.Value = "O'Hara"
com.Parameters!pblob.Value = bin
com.ExecuteNonQuery()
Tip

The definition of a parameter and specification of the parameter value for the first call can be squeezed into a single line:

com.Parameters.Add("ptext", OdbcType.Text).Value = "O'Hara"

The equivalent C# code looks like this:

OdbcCommand com = new OdbcCommand();
com.Connection = odbcconn;
com.CommandText =
    "INSERT INTO tablename (a_date, a_text, a_blob) VALUES(?, ?, ?)";
com.Parameters.Add("pdate", OdbcType.DateTime);
com.Parameters.Add("ptext", OdbcType.Text);
com.Parameters.Add("pblob", OdbcType.Binary);
byte[] bin = new byte[50];
   . . .
com.Parameters["pdate"].Value = DateTime.Now;
com.Parameters["ptext"].Value = "O'Hara";
com.Parameters["pblob"].Value = bin;
com.ExecuteNonQuery();

Parameters with MySQL Providers

In the use of parameters, there are considerable differences between the ODBC provider and the various MySQL providers. There, you must pass the parameters in the form @parametername. (This has the advantage that the parameters cannot be confused among one another, and thus the order of Parameters.Add instructions is arbitrary.) The following lines were tested with the ByteFX provider:

Dim com As New MySQLCommand()
com.Connection = myconn
com.CommandText = _
  "INSERT INTO tablename (a_date, a_text) VALUES(@p1, @p2)"
com.Parameters.Add("@p1", MySQLDbType.Datetime)
com.Parameters.Add("@p2", MySQLDbType.String)
com.Parameters("@p1").Value = Now
com.Parameters("@p2").Value = "O'Hara"
com.ExecuteNonQuery()

Reading SELECT Results (OdbcDataReader)

The OdbcDataReader class offers the simplest way of reading SELECT results. The OdbcDataReader does not support free navigation in the result list (hence only forward only). Nor are changes possible. In return, OdbcDataReader is frugal with resources and offers the most efficient path to SELECT results.

You obtain an OdbcDataReader when you execute an SQL command with the method ExecuteReader. Read reads the first, or next, record, and returns True or False depending on whether a record was found. The OdbcDataReader object should be explicitly closed with Close as soon as it is no longer needed. (Otherwise, it holds onto ODBC resources.)

The following lines show how all records can be run through in a loop:

Dim com As New OdbcCommand( _
  "SELECT publID, publName FROM publishers", odbcconn)
Dim dr As OdbcDataReader = com.ExecuteReader()
While dr.Read()
  Console.WriteLine("id: {0} name: {1}", dr!publID, dr!publName)
End While
dr.Close()

The OdbcDataReader offers no way of determining the number of records in advance. Thus you must count along as you read all the records, or else execute a SELECT COUNT(*) ... query.

Access to data fields: In VB.NET, access to data fields takes place in the form dr!columnname. In C#, you must instead write dr["columnname"]. In each case, you obtain as result an object of the interface IDataRecord, which you must explicitly transform when saving into another data type.

As an alternative, you can read data with GetByte(n), GetDateTime(n), GetInt32(n), etc., where n is the column number (0 for the first column).

NULL: Data fields that in the database contain NULL have in ADO.NET the value System.DBNull.A NULL test can be caried out with the method Convert.IsDBNull:

If Convert.IsDBNull(dr!column) Then ...

In saving SELECT results (e.g., stringvar = CStr(dr!textcolumn)), you must always consider the case that the data field contains NULL: Then the VB.NET method CStr or the C# cast operator (string) leads to an error.

If you wish to save the value NULL,you can prefix the parameter DBNull.Value:

com.Parameters(0).Value = DBNull.Value

Dates: Dates (DATE, TIMESTAMP) are automatically transformed in ADO.NET into Date objects.

Times: Times (TIME)are transformed in ADO.NET into TimeSpan objects.

Fixed-point numbers: Fixed-point numbers (DECIMAL) are represented in ADO.NET by Decimal objects.

BIGINT s: In contrast to ADO, ADO.NET has no problems with 64-bit integers (BIGINT). Such numbers are represented in ADO.NET as Long numbers.

BLOB s: Binary data are represented in ADO.NET by Byte fields.

Character strings: In VB.NET and C#, character strings are coded in UTF16. In transfer between ADO.NET and MySQL, a transformation takes place automatically into the character set of the MySQL server (generally latin1). Unrepresentable characters are replaced by question marks without an error message.

If you wish to save Unicode strings as such in MySQL, you must use a BLOB field. To transform a .NET String, say s, into a UTF8 Byte field, use Text.Encoding.UTF8.GetBytes(s). To transform in the opposite direction, use Text.Encoding.UTF8.GetString(bytes). The following lines show how a Unicode string is stored and then read:

com.CommandText = "INSERT INTO table (a_blob) VALUES (?)"
com.Parameters.Add("blob", OdbcType.Binary)
com.Parameters!blob.Value = _
  Text.Encoding.UTF8.GetBytes("Smiley: " +  ChrW(9786))
com.ExecuteNonQuery()
com.CommandText = "SELECT * FROM table"
Dim dr As OdbcDataReader = com.ExecuteReader()
dr.Read()
Dim s As String = _
  Text.Encoding.UTF8.GetString(CType(dr!a_blob, Byte()))

Evaluating SELECT Scalars (OdbcCommand.ExecuteScalar)

Queries such as SELECT COUNT(*) FROM ... represent a special case. They return a single record with a single column. In this case, you can do without a DataReader. Instead, execute the SQL command (OdbcCommand object) with the method ExecuteScalar, which returns the result as an Object. With CInt (C# cast operator int), you obtain the corresponding Integer:

Dim n As Integer
Dim com As New OdbcCommand("SELECT COUNT(*) FROM table", odbcconn)
n =  CInt(com.ExecuteScalar())

AUTO_INCREMENT Columns (LAST_INSERT_ID)

The ID number of the last data record inserted with INSERT can be determined with the command SELECT LAST_INSERT_ID( ).The method ExecuteScalar,just described, returns the ID number as a long integer:

Dim n As Long
Dim com As New OdbcCommand()
com.Connection = odbcconn
com.CommandText = "INSERT INTO table (columns  . . .    )  VALUES (ldots)"
com.ExecuteNonQuery()
com.CommandText = "SELECT LAST_INSERT_ID()"
n =  CLng(com.ExecuteScalar())
DataSet, DataTable, OdbcDataAdapter

If you wish to navigate at will (backward as well as forward) through SELECT results and alter, delete, or insert new records, then you need a DataSet object and at least one DataTable contained therein. (A DataSet helps in managing one or more DataTables. A DataTable is simply a table with data, usually the result of SELECT queries. DataSet and DataTable are general ADO.NET classes, and are independent of the provider. For this reason, there are no independent OdbcDataSets, MySqlDataSets, etc. However, for communication between the DataSet and the provider, an OdbcDataAdapter or MySqlDataApapter is necessary.)

All the data contained in a DataSet are located in the memory of the ADO.NET program. When a DataTable contained in a DataSet is created on the basis of a SELECT query, all resulting records from the query are transmitted to the ADO.NET program. From this point on, you are processing the local copy of the data. The data stored on the MySQL server are updated only when the Update method for the OdbcDataAdapter is executed.

The following commands show how to fill a DataTable with the results of a SELECT query. As usual, you first create an OdbcCommand object for your SQL command. This object will be passed to an OdbcDataAdapter. The SELECT query is executed by the method Fill of the OdbcDataAdapter. The results are stored in the DataSet. For this, a new DataTable is created in the DataSet, which in this example is given the name "test".You can now access this DataTable via ds.Tables("name"):

Dim com As New OdbcCommand("SELECT * FROM testall", odbcconn)
Dim da As New OdbcDataAdapter(com)
Dim ds As New DataSet()                 'preveiously empty DataSet ds
da.Fill(ds, "test")                     'creates DataTable "test" in ds
Dim dt As DataTable = ds.Tables("test") 'access to the DataTable

You can now access the data of the DataTable in the form of dt.Row(rowno)!columnname.Here, Row(n) return a single record (class DataRow), and Rows.Count gives the number of records. A loop over all records can be formulated thus:

Dim row As DataRow
For Each row In dt.Rows   'loop over all records
    row!columnname =  . . .           'change data
Next

With datarow.Delete you can delete individual records. It is somewhat more complex to insert new records. For that, you first create a new record with datatable.NewRow, change its contents, and finally add it with datatable.Rows.Add to the DataTable:

Dim newrow As DataRow
newrow = dt.NewRow()
newrow!a_big_int = 12345
dt.Rows.Add(newrow)

As already mentioned, all changes in the DataSet are valid initially only locally (that is, in memory of the ADO.NET program). For the changes to be carried out in the MySQL database, you must execute the method Update for the OdbcDataAdapter. The update is carried out in a series of INSERT, UPDATE, and DELETE commands, which the OdbcDataAdapter sends to MySQL. However, the OdbcDataAdapter is not able to generate these commands. Therefore, first, before the Update method, an OdbcCommandBuilder object must be generated to take over this task. The following lines demonstrate how that is done:

'  . . .    Code to alter data in dt
Dim cb As New OdbcCommandBuilder(da)
Try
    da.Update(ds, "test")
Catch e As Exception
    Stop
End Try

If you wish to know which UPDATE, INSERT, and DELETE commands are being executed, you can find this out with cb.GetInsert/Update/DeleteCommand( ). (These commands depend heavily on the provider being used.)

What looks great in theory, is in practice hemmed in with many limitations:

  • Changes can be stored only if the DataTables contained in the DataSet come directly form a MySQL table. DataTables based on SELECT commands that include several tables (JOIN) or use aggregating functions (GROUP BY ) cannot be updated. In this case, the OdbcCommandBuilder is incapable of determining the appropriate SQL commands. (This limitation is not one of ADO.NET, but has to do with the fact that changes in such tables can often be interpreted in more than one way, and ADO.NET and the CommandBuilder are not clairvoyant.)

  • In general, the change in a DataSet is strongly provider-dependent. My worst experiences were (not surprisingly) with the ODBC provider. The MySQL providers, on the other hand, are better adapted to the peculiarities of MySQL. For example, after the insertion of a new record, the ByteFX provider automatically determines the ID number of the new record (AUTO_INCREMENT column) and updates the row in the DataTable accordingly. The ODBC variant is incapable of doing this, which makes the insertion of linked records into several tables of a DataSet impossible from the outset.

    The strong provider-dependence also means that you cannot count on your code functioning after a change of provider.

  • Aside from these limitations, in my experiments with DataSet updating, various other problems and errors arose, whose cause was not always completely discernible. (The generally cryptic error messages were not very helpful. Some of the errors perhaps have to do with incorrect handling of MySQL-specific data types.)

In sum, not everything that looks technologically appealing is ready to be used in the real world. I recommend that you use DataSets in general only for reading data. If you wish to alter data, you should create the necessary SQL commands yourself and execute them with OdbcCommands (preferably using parameters).

Windows.Forms and ASP.NET Controls

The .NET Framework offers a great number of controls for Windows and Internet applications (ASP.NET) that can be bound to DataSets. This simplifies the representation and processing of lists and tables on the basis of query results.

A basic rule is that as long as you display only data with the controls, and don't try to alter the data, you may expect few problems. However, if you permit data updating, then you should test extensively what is possible. (Internally, data bound controls use DataSets almost exclusively, and I have already referred to their Update problems.)

Space here does not suffice to introduce data bound controls and go into the details of their deployment. The following example program should at least provide a first impression of what can be done with them. The program contains a listbox with all publishers. When one is selected, a table field displays that publisher's titles (see Figure 17-14):

'  example program vbnet\datacontrols
Imports Microsoft.Data.Odbc
Public Class Form1
Inherits System.Windows.Forms.Form
'  Windows Forms Designer generated code  . . .
Dim odbcconn As OdbcConnection
Dim titlecommand As OdbcCommand
'  initialization tasks
Private Sub  Form1_Load( _
  ByVal sender As System.Object, _
  ByVal e As System.EventArgs) Handles MyBase.Load
  '  establish connection to the database
  odbcconn = New OdbcConnection( _
"Driver={MySQL ODBC 3.51 Driver};Server=localhost;" + _
"Database=mylibraryodbc;UID=root;PWD=saturn;Options=3")
  odbcconn.Open()
  'prepare OdbcCommand for titles
  titlecommand = New OdbcCommand( _
"SELECT title, subtitle, langName FROM titles, languages " + _
"WHERE publID = ?  AND titles.langID = languages.langID " + _
"ORDER BY title", _
odbcconn)
  titlecommand.Parameters.Add("publID", OdbcType.Int)
  'fill the listbox for publishers
  Dim com As New OdbcCommand( _
"SELECT publID, publName FROM publishers ORDER BY publName", _
odbcconn)
  Dim da As New OdbcDataAdapter(com)
  Dim ds As New DataSet()
  da.Fill(ds, "publishers")
  ComboBox1.DisplayMember = "publName"
  ComboBox1.ValueMember = "publID"
  ComboBox1.DataSource = ds.Tables("publishers")
End Sub
'  update table fields for each selection in the listbox
Private Sub ComboBox1_SelectedIndexChanged( _
  ByVal sender As System.Object, _
  ByVal e As System.EventArgs) _
  Handles ComboBox1.SelectedIndexChanged
  titlecommand.Parameters("publID").Value = ComboBox1.SelectedValue
  Dim ds As New DataSet()
  Dim da As New OdbcDataAdapter(titlecommand)
  da.Fill(ds, "titles")
  DataGrid1.DataSource = ds.Tables("titles")
    End Sub
    End Class
Click To expand
Figure 17-14: ADO.NET example program

Transactions

At the beginning of a transaction, execute the method BeginTransaction for the object OdbcConnection object. As result, you obtain an OdbcTransaction object, with whose method Commit or Rollback you can end the transaction. For all commands that are executed within the confines of the transaction, the Transaction property must be set, either through the third parameter of the OdbcCommand constructor or through an explicit allocation of the Transaction property. The following lines show the usual way of proceeding:

Dim tr As OdbcTransaction
tr = odbcconn.BeginTransaction()
Dim com As New OdbcCommand("UPDATE  . . .    ", odbcconn, tr)
com.ExecuteNonQuery()
'  . . .    further commands of the transaction
tr.Commit()
Remark

The InnoDB documentation recommends in client programming not to rely in general on the transaction methods of the particular library or API, and instead, to execute the SQL commands SET AUTOCOMMIT= ... and BEGIN/COMMIT/ROLLBACK explicitly. The reason is that with many libraries, the transaction functions have not been sufficiently tested in combination with MySQL. In my experience, I can say at least that with ADO.NET in combination with the ODBC driver, everything seems to work without any problems.

Example: Storing New Titles in mylibraryodbc

The example program of this section consists of a simple input mask in which you can input new book titles together with their authors and publishers. In the author field you may input more than one name, separated by semicolons. The publisher field is allowed to remain empty (in which case, the title is stored without a publisher). (See Figure 17-15.) In storing a new title, a check is made whether the authors and publisher are already known to the database, in which case the records already stored are used. There are no other plausibility checks or error-checking.

Click To expand
Figure 17-15: Form for storing a new book title

The program code is broken into two procdures. In Form1_Load, a number of OdbcCommand objects are initialized with INSERT and SELECT commands. These commands are then used when the procedure btnSave_Click is called for storing input:

'  example vbnet\newtitleform
Option Strict On
Imports Microsoft.Data.Odbc
Public Class Form1
  Inherits System.Windows.Forms.Form
  '  Windows Forms Designer generated code  . . .
  Dim odbcConn As OdbcConnection
  Dim insertPublisherCom, insertAuthorCom, _
  insertTitleCom, insertRelAuthTitleCom, _
  selectPublisherCom, selectAuthorCom, _
  lastIDCom As OdbcCommand
  Private Sub Form1_Load(  . . .    )  Handles MyBase.Load
    '  establish connection to the database
    odbcconn = New OdbcConnection( _
  "Driver={MySQL ODBC 3.51 Driver};Server=localhost;" + _
  "Database=mylibraryodbc;UID=root;PWD=saturn;Options=3")
    odbcconn.Open()
    'prepare various INSERT and SELECT commands
    insertAuthorCom = New OdbcCommand( _
  "INSERT INTO authors (authName) VALUES (?)", odbcConn)
    insertAuthorCom.Parameters.Add("name", OdbcType.Text)
    selectAuthorCom = New OdbcCommand( _
  "SELECT authID FROM authors WHERE authName = ?  LIMIT 1", _
  odbcConn)
    selectAuthorCom.Parameters.Add("name", OdbcType.Text)
    insertTitleCom = New OdbcCommand( _
  "INSERT INTO titles (title, publID) VALUES (?, ?)", _
  odbcConn)
    insertTitleCom.Parameters.Add("title", OdbcType.Text)
    insertTitleCom.Parameters.Add("publID", OdbcType.Int)
    insertPublisherCom = New OdbcCommand( _
  "INSERT INTO publishers (publName) VALUES (?)", odbcConn)
    insertPublisherCom.Parameters.Add("name", OdbcType.Text)
    selectPublisherCom = New OdbcCommand( _
  "SELECT publID FROM publishers WHERE publName = ?  LIMIT 1", _
  odbcConn)
    selectPublisherCom.Parameters.Add("name", OdbcType.Text)
  insertRelAuthTitleCom = New OdbcCommand( _
    "INSERT INTO rel_title_author (titleID, authID) " + _
    "    VALUES (?, ?)", odbcConn)
  insertRelAuthTitleCom.Parameters.Add("titleID", OdbcType.Int)
  insertRelAuthTitleCom.Parameters.Add("authID", OdbcType.Int)
  lastIDCom = New OdbcCommand( _
    "SELECT LAST_INSERT_ID()", odbcConn)
End Sub
Private Sub btnSave_Click(  . . .    )  Handles btnSave.Click
  Dim publID, titleID As Integer
  Dim result As Object
  Dim author, authors() As String
  txtTitle.Text = Trim(txtTitle.Text)
  txtAuthors.Text = Trim(txtAuthors.Text)
  txtPublisher.Text = Trim(txtPublisher.Text)
  If txtTitle.Text = "" Or txtAuthors.Text = "" Then
    MsgBox("Please specify title and authors!")
    Exit Sub
  End If
  'search for and store publisher in publisher table
  If txtPublisher.Text <> "" Then
    '  does the publisher already exist?
    selectPublisherCom.Parameters(0).Value = txtPublisher.Text
    result = selectPublisherCom.ExecuteScalar()
    If result Is Nothing Then
  'no: store
  insertPublisherCom.Parameters(0).Value = txtPublisher.Text
  insertPublisherCom.ExecuteNonQuery()
  publID = CInt(lastIDCom.ExecuteScalar())
    Else
  publID = CInt(result)
    End If
  End If
  '  store title
  insertTitleCom.Parameters(0).Value = txtTitle.Text
  If publID > 0  Then
    insertTitleCom.Parameters(1).Value = publID
  Else
    insertTitleCom.Parameters(1).Value = DBNull.Value
  End If
  insertTitleCom.ExecuteNonQuery()
  titleID = CInt(lastIDCom.ExecuteScalar())
      'store authors
  authors = Split(txtAuthors.Text, ";")
  insertRelAuthTitleCom.Parameters(0).Value = titleID
  For Each author In authors
    'does the author already exist?
    selectAuthorCom.Parameters(0).Value = author
    result = selectAuthorCom.ExecuteScalar()
    If result Is Nothing Then
  '  no, store new author
  insertAuthorCom.Parameters(0).Value = author
  insertAuthorCom.ExecuteNonQuery()
  insertRelAuthTitleCom.Parameters(1).Value = _
      CInt(lastIDCom.ExecuteScalar())
    Else
  insertRelAuthTitleCom.Parameters(1).Value = _
    CInt(result)
  End If
    '  store link title/authors
    insertRelAuthTitleCom.ExecuteNonQuery()
  Next
  MsgBox("Your input has been saved")
  txtTitle.Text = ""
  txtAuthors.Text = ""
  txtPublisher.Text = ""
End Sub


Converter: Microsoft SQL Server ® MySQL

If up to now you have managed a database with the Microsoft SQL Server (or the MSDE, that is, the Microsoft Data Engine) and are considering porting it to MySQL, there are two main ways of bringing your data into a MySQL database.

  • Summon Access to your assistance. First, import the table into an empty Access database, and then export it to MySQL.

  • Use the VBA/Visual Basic script mssql2mysql (developed in its first version by me). The program should generally enable a more exact copy of the database structure, since there is no intermediate Access step. The remainder of this section is related to this program. The program can also be seen as a further example of ADO database programming with MySQL, although for reasons of space we omit the source code.

    Pointer

    Converters, both commercial and free of charge, between MySQL and a number of other databases (dBase, FoxPro, mSQL) can be found at the following addresses:

    http://www.mysql.com/portal/software/index.html
http://www.mysql.com/doc/en/Contrib.html

Properties of mssql2mysql

  • freely obtainable (GPL)

  • copies an entire database, that is, all user-defined tables (structure, indexes, data)

  • automatically changes the name of a table or column if the name is not allowable in MySQL (for example, My table becomes My_table)

  • changes Unicode character strings into ANSI character strings or BLOBs

  • generates either a text file with SQL commands or executes the required commands directly on the MySQL server

Assumptions

  • To execute the script, either a more or less current VBA interpreter (e.g., in all components of Microsoft Office since Office 2000) or the programming language Visual Basic 6 must be available.

  • The program requires the libraries ADO and SQLDMO. (SQLDMO allows the control of the Microsoft SQL server. The library is installed together with the SQL server.)

  • If the converted database is not to be written into a text file (*.sql), but transmitted directly to a running MySQL server, then Connector/ODBC must also be installed.

The converter was tested with SQL Server 7 and 2000, MySQL 3.23 and 4.0, ADO 2.5 and 2.7, and Connector/ODBC 2.50 and 3.51.n.

Restrictions

  • In MySQL, currently unavailable or incompatible (with the SQL server) database attributes are not supported (views, stored procedures, user-defined data type, access privileges, etc.).

  • MySQL allows only one AUTO_INCREMENT column per table, and that column must be defined as PRIMARY KEY. The converter does not check this condition. You must therefore change your SQL server as necessary before starting the script.

  • SQL server TIMESTAMPs are incompatible with MySQL TIMESTAMPs and are therefore transformed into TINYBLOBs.

  • Integrity rules (foreign key constraints) are not supported by the script.

How to Use It

The text file mssql2mysql is available over the Internet:

http://www.kofler.cc/mysql/mssql2mysql.html

To execute the code, use either Visual Basic 6 or a program with VBA 6 support (for example, Excel 2000, Word 2000, Access 2000).

  • Visual Basic 6: If you are working with Visual Basic 6, begin a new standard project and insert the entire code into the code window of a form. Then start the program with F5.

  • VBA: If you are working with a VBA 6 compatible program, switch with Alt+F11 into the VBA development environment, insert a new, empty, module, and copy the code into this module. Then with F5 open the dialog for macro execution and execute the procedure main.

Setting Parameters

All the parameters of the conversion process are controlled by a number of constants, which can be found at the beginning of the code. The first five constants describe the database on the Microsoft SQL server. The login to the database can be effected either via the security system integrated into Windows NT/2000/XP or by the explicit specification of user name and password:

Const MSSQL_SECURE_LOGIN = True    'login type (True for NT Security)
Const MSSQL_LOGIN_NAME = ""        'username ("" for NT Security)
Const MSSQL_PASSWORD = ""          'password (""for NT Security)
Const MSSQL_HOST = "mars"          'computer ("(local)" for localhost)
Const MSSQL_DB_NAME = "pubs"       'database name

OUTPUT_TO_FILE specifies whether the program should generate a text file with SQL commands or whether the commands should be executed at once. The first variant has the advantage that if problems arise, the text file can be directly edited:

Const OUTPUT_TO_FILE = 0         '1 write file,
      '0 execute SQL commands at once

If you have decided for OUTPUT_TO_FILE=1, you must specify the name of the resulting file in OUTPUT_FILENAME:

Const OUTPUT_FILENAME = "c:\export.sql"

With OUTPUT_TO_FILE=0 you must specify all parameters so that a connection to MySQL can be created. For this there must be sufficient access privileges so that CREATE DATABASE, etc., can be executed:

Const MYSQL_USER_NAME = "root"    'username
Const MYSQL_PASSWORD = "uranus"   'password
Const MYSQL_HOST = "localhost"    'computer name or "localhost"
Const MYSQL_PORT = 3306           'MySQL port

Which of the following two commands is necessary depends on whether you are using Connector/ODBC 3.51 or 2.51:

Const MyODBCVersion = "MySQL ODBC 3.51 Driver"
Const MyODBCVersion = "MySQL" 'for 2.50.n

Finally, we mention a few conversion options:

  • NEW_DB_NAME enables the name of the new MySQL database to be given. If the constant remains empty, the new database keeps the same name as the SQL server database.

  • UNICODE_TO_BLOB specifies whether UNICODE character strings should be transformed into BLOBs (True) or ANSI character strings (False). The second variant makes sense if Unicode data types were used in the SQL server database, but within them no special characters were used outside of the ANSI character set.

  • DROP_DATABASE specifies whether at the beginning of the conversion any existing MySQL database with the same name should be deleted.

  • MAX_RECORDS specifies how many records per table should be converted. Here 0 means that all data records should be converted. This option is practical for first carrying out a quick test and, for example, converting only ten records per table. Only after it has been verified that all is well is the entire database converted:

    Const NEW_DB_NAME = ""         'MySQL database name
Const UNICODE_TO_BLOB = False  'Unicode --> BLOBs?
Const DROP_DATABASE = True     'begin with  DROP database?
Const MAX_RECORDS = 0          '0: convert all data

Graphical User Interface for mssql2mysql

If you have VB6, you can also use a graphical user interface to mssql2mysql, developed by Roberto Alicata based on the script version 0.06 (see Figure 17-16). This version exhibits no new functions, but it is intuitively easy to use.

Click To expand
Figure 17-16: Graphical user interface for mssql2mysql


Part IV: Reference

Chapter List

Chapter 18: SQL Reference
Chapter 19: MySQL Tools
Chapter 20: API Reference


Chapter 18: SQL Reference

Overview

This chapter gives an overview of the SQL operators, functions, and commands available under MySQL. My goal in organizing the information in this chapter was to give you, dear reader, a compact overview of the most important and useful syntactic variants.

Please note that MySQL boasts countless extensions as well as, alas, certain shortcomings with respect to the ANSI-SQL/92 standard.

This chapter is based on MySQL 4.0. To the extent that they are known, new syntactic features of MySQL 4.1 are also described. (We will always make clear what information relates specifically to version 4.1.) Appendix B contains a summary of what is new in MySQL 4.1.


Syntax

We begin with a brief section describing the syntax of object names, character strings, dates and times, and binary data.

Object Names

Names of objects—databases, tables, columns, etc.—can be at most 64 characters in length. Permitted characters are all the alphanumeric characters of the character set used by MySQL as well as the characters _ and $. There are two reasons for this:

  • The coding of special characters depends on the character set in force in MySQL and the client program. If these settings are changed, then access to objects might become problematic.

  • The names of databases and tables can be stored in files, and it is not MySQL, but the operating system that is responsible for the naming rules for files. This can be yet another source of conflict (especially if databases must be exchanged among various operating systems).

For reasons of practicality, it is a good idea to limit oneself to the alphanumeric ASCII characters and the underscore, and generally to avoid the use of special characters.

Table names that do not refer to the current database must have the database name prefixed to them. Likewise, the name of a column must be extended by the name of the table and that of the database if the column name alone fails to provide a unique identification (such as in queries in which several like-named columns in different tables appear):

  • Table names: tablename or db.tablename

  • Column names: colname or tblname.colname or dbname.tblname.colname

Case Sensitivity

The following objects are listed according to whether they exhibit case sensitivity:

  • Case Sensitivity: Database names, table names, alias names, variable names

  • No Case Sensitivity: SQL commands and functions, column names, index names

Under Windows, MySQL is flexible with respect to case in the naming of databases and tables. The reason is that the operating system does not distinguish case in the naming of directories and files. Note, however, that case must be consistent within an SQL command. The following command will not function properly: SELECT * FROM authors WHERE Authors.authName = "xxx".

Since MySQL 4.0.2 and 4.1.1, MySQL under Windows uses exclusively lowercase names in the creation of new databases and tables (regardless of how it is written in the CREATE command). This should simplify the migration of databases from

Windows to Unix/Linux. This automatic transformation is the result of the option lower_case_table_names, which is set to 1 under Windows by default.

Character Strings in MySQL 3.23 and 4.0

MySQL 4.0 offers good support for 8-bit character sets. Multibyte character sets like Unicode will be correctly supported only starting with MySQL 4.1; see the next section. Character strings can be enclosed in single or double quotes. The following two expressions are equivalent in MySQL (though only the single-quote variant conforms to the ANSI-SQL/92 standard):

'character string' "character string"

If a quotation mark should happen to be part of the character string, then there are various ways of expressing this:

"abc'abc"    means    abc'abc
"abc""abc"   means    abc"abc
"abc\'abc"   means    abc'abc
"abc\"abc"   means    abc"abc
'abc"abc'    means    abc"abc
'abc''abc'   means    abc'abc
'abc\"abc'   means    abc"abc
'abc\'abc'   means    abc'abc

Within a character string, the special characters provided by the prevailing character set are allowed, for example, äöüßáàé˝o if you are working with the default character set ISO-8859-1 (latin1). However, some special characters must be specially coded:

Quoting of Special Characters Within a Character String
\0 | 0character (code 0)
\b | backspace character (code 8)
\t | tab character (code 9)
\n | newline character (code 10)
\r | carriage return character (code 13)
\" | double quote character (code 34)
\' | single quote character (code 39)
\\ | backslash (code 92)

If x is not one of the above-mentioned special characters, then \x simply returns the character x. Even if a character string is to be stored as a BLOB (binary object), then the 0 character as well as the single quote, double quote, and backslash must be given in the form \0, \', \", and \\.

Instead of indicating special characters in character strings or BLOBs by the backslash escape character, it is often easier simply to specify the entire object using hexadecimal notation. MySQL accepts hex codes of arbitrary length in SQL commands in the following form: 0x4142434445464748494a.

However, MySQL is incapable of returning the result of a query in this form. (If you are working with PHP, then that programming language offers a convenient function for this purpose: bin2hex.)

Pointer

If two character strings are to be concatenated, then you must use the function CONCAT. (The operator + will not serve the purpose.) In general, MySQL provides a broad range of functions for working with character strings.

Character Strings in MySQL 4.1

Starting with MySQL 4.1, three things will change with respect to character strings:

  • The character string and sort order (collation) will be set independently of each other.

  • Each table, and indeed each column within a table, can have its own character set and its own sort order.

  • Unicode is now a possible choice for the character set (in formats UTF8 and UCS2 = UTF16).

This has consequences, of course, for the use of character strings in SQL commands. One can specify the character set for every string. For this, you can use the function CONVERT or the cast operator _characterset. Here are two equivalent examples, demonstrating the internal Unicode encoding in the format UTF8:

SELECT HEX(CONVERT('ABCäöü' USING utf8))

414243C3A4C3B6C3BC

SELECT HEX(_utf8 'ABCäöü ')

414243C3A4C3B6C3BC

In many cases, it can be necessary to specify the sort order as well as the character set. (This determines which characters are considered equivalent and how character strings are to be sorted.) For this, one has the syntax _characterset 'abc' COLLATE collname.Here is an example:

SELECT _latin1 'a' = _latin1 'ä'

0

SELECT _latin1 'a' COLLATE german1 = _latin1 'ä' COLLATE german1

1

Pointer

Further details on string management in MySQL 4.1 and other innovations in this version can be found in Appendix B.

Numbers

Decimal numbers are written with a period for the decimal point and without a thousands separator (thus 27345 or 2.71828). One may also use scientific notation (6.0225e23 or 6.626e-34) for very large or very small numbers.

MySQL can also interpret hexadecimal numbers prefixed by 0x.Depending on the context, the number is interpreted as a character string or as a 64-bit integer:

SELECT 0x4142434445464748494a

ABCDEFGHIJ

SELECT 0x41 + 0

66

Automatic Transformation of Numbers and Character Strings

In carrying out an operation on two different data types, MySQL makes every attempt to find a compatible data type. Integers are automatically changed into floating-point numbers if one of the operators is a floating-point number. Character strings are automatically changed into numbers if the operation involves a calculation. (If the beginning of the character string cannot be interpreted as a number, then MySQL calculates with 0.)

SELECT '3.14abc' + 1

4.14

Date and Time

MySQL represents dates as character strings of the form 2003-12-31, and times in the form 23:59:59. With the data type DATETIME both formats are simply concatenated, yielding, for example, 2003-12-31 23:59:59. In dealing with values in TIMESTAMP columns, MySQL leaves out all spaces, hyphens, and colons. In our example, we would have 20031231235959:

USE exceptions
SELECT * FROM test_date id
 id a_date     a_time   a_datetime          a_timestamp
--------------------------------------------------------------------------
 1  2003-12-07 09:06:29 2003-12-07 09:06:29 20031207090649
Caution

The default setting of TIMESTAMP has changed in MySQL 4.1. It is now returned from the server in the format YYYY-MM-DD HH:MM:DD, that is, like DATETIME values. Add a zero if you wish to use the oldformat(SELECT ts+0 FROM table).

In storing dates, MySQL is quite flexible: Both numbers (e.g., 20031231) and character strings are accepted. Hyphens are allowed in character strings, or they can simply be done without. If a year is given but no century is specified, then MySQL automatically uses the range 1970–2069. Therefore, MySQL accepts the following character strings for a DATETIME column: '2003 12 31', '20031231', '2003.12.31', and '2003&12&31'.

Remark

In storing dates, MySQL carries out only an incomplete check as to whether the date is a possible one. Thus, for example, it is possible to store '2003-2-29' in a date field, even though 2003 was not a leap year.

The date '0000-00-00' has a special meaning. This value is officially allowed as a date in MySQL. MySQL uses this value itself when it recognizes a date as patently incorrect (e.g., '2003-2-32').

Binary Data

Binary data that are to be stored in BLOB fields are dealt with in SQL commands like character strings; see above. (However, there are differences in sorting.)

Comments

There are three ways of supplying comments in SQL commands:

SELECT 1 # comment
SELECT 1 /* comment */
SELECT 1 -- comment since MySQL 3.23

Comments that begin with # or with -- (there must be a space after the --) hold until the end of the line. Comments between /* and */ can extend over several lines, as in C. Nesting is not allowed.

If youwish to write SQL code that makes use of some of the peculiarities of MySQL yet remains compatible as much as possible with other dialects, a particular variant of the comment is often useful:

SELECT /*! STRAIGHT_JOIN */ col FROM table ...

With the MySQL-specific SELECT extension, STRAIGHT_JOIN will be executed only by MySQL; all other SQL dialects will consider this a comment.

Avariant of this enables differentiation among various MySQL dialects:

CREATE /*!32302 TEMPORARY */ TABLE ...

In this case, the key word TEMPORARY is processed only if the command is executed by MySQL 3.23.02 or a more recent version.

Semicolons at the End of SQL Commands

Caution

Be careful with semicolons! On the basis of your experience with the MySQL monitor (that is, the program mysql), you have probably become accustomed to placing a semicolon at the end of SQL commands. Even phpMyAdmin deals properly with such semicolons (in particular, to separate several SQL commands from each other). However, the following must be observed: Neither ANSI-SQL nor the SQL dialect of MySQL allows semicolons at the end of an instruction or for separating commands!

Semicolons are thus merely a peculiarity of mysql and phpMyAdmin. If you send SQL commands to MySQL in other programs (or in program code), they must appear without semicolons. Likewise, it is generally not permitted to send several commands at once. If you wish to execute several commands, you must send them one at a time (e.g., in a loop).


Operators

MySQL Operators

 | Arithmetic Operators
+, -, *, / | basic arithmetic
% | mod (remainder of integer division)
DIV | alternative division operator (since MySQL 4.1)
MOD | alternative modulo operator (since MySQL 4.1)
 | BitOperators
| | binary OR
& | binary AND
˜ | binary negation (inverts all bits)
<< | shift bits left (multiplication by 2n)
>> | shift bits right (division by 2n)
 | Comparison Operators
= | equality operator
<=> | equality operator, permitting comparison with NULL
!=, <> | inequality operator
<, >, <=, >= | comparison operators
IS [NOT] NULL | NULL comparison
BETWEEN | range membership (e.g., x BETWEEN 1 AND 3)
IN | set membership (e.g., xIN (1, 2, 3) or xIN('a', 'b', 'c'))
NOT IN | set membership (e.g., xNOT IN ('a', 'b', 'c'))
 | Pattern Matching
[NOT] LIKE | simple pattern matching (e.g., x LIKE 'm%')
[NOT] REGEXP | extended pattern matching (e.g., x REGEXP '.*x$')
SOUNDS LIKE | corresponds to SOUNDEX(a) = SOUNDEX(b),since MySQL 4.1
 | Binary Comparison
BINARY | identifies operand as binary (e.g., BINARY x = y)
 | Logical Operators
!,NOT | negation
||,OR | logical OR
&&,AND | logical AND
XOR | logical exclusive OR (new in MySQL 4.0)
 | Cast Operator (since MySQL 4.1)
_charset 'abc' | for the string 'abc' the character set charset is valid
_chs 'abc'COLLATE coll | string 'abc' has character set charset and sortorder coll

Arithmetic Operators, Bit Operators

Arithmetic operators for which one of the operands is NULL generally return NULL as result. In MySQL, a division by zero also returns the result NULL.

Comparison Operators

Comparison operators normally return 1 (corresponding to TRUE) or 0 (FALSE). Comparisons with NULL return NULL. The two exceptions are the operators <=> and IS NULL, which even in comparison with NULL return 0 or1:

SELECT  NULL=NULL, NULL=0
  NULL, NULL
SELECT  NULL<=>NULL, NULL<=>0
  1, 0
SELECT  NULL IS NULL, NULL IS 0
  1, 0

In comparing character strings with <, <=, >, >=, as well as with BETWEEN (and, of course, with all sorting operations), the character set in force when MySQL was launched plays a decisive role. In the default setting, the character set is latin1 (ISO-8859-1) with Swedish sort order.

The following examples show that the default behavior of MySQL is not always what you would expect. Thus, for example, UCASE (for changing characters to uppercase) does not apply to all special characters. Character comparison, however, works much better:

SELECT UCASE('abc äöüÿ åêõ')
    ABC Äöüÿ Åêõ
SELECT  'a'='A', 'a'='ä', 'ä'='Ä', 'ö'='Ö', 'ü'='Ü'
'a'='A' 'a'='ä' 'ä'='Ä' 'ö'='Ö' 'ü'='Ü'
----------------------------------------------------
   1       0       1       1       1

In sorting, Swedish rules are in effect. Here we do not display the result of the query by columns in order to save space. Various special characters and unrepresentable characters have been omitted from the result (and replaced by …):

USE exceptions
SELECT a_char FROM  test_order_by
WHERE id>31
ORDER BY  a_char
!, ", #, $, %, &, ', (, ), *, +, ,, -, ., /, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, :, ;, <,
=, >, ?, @, A, a, À,
Ê, Ë, è, é, ê, ë, F, f, G, g, H, h, I, i, Ì, Í, Î, Ï, ì, í, î, ï, J, j, K, k, L, l,
M, m, N, n, Ñ,
t, U, u, Ù, Ú, Û, ù, ú, û, V, v, W, w, X, x, Y, y, Ü,  Y, ü,       Z, z, [, Å,
å, \, Ä, Æ, ä, æ, ], Ö, ö,    ^, _, `, {, |, }, ~, …,  f , … , S,   ‹, Œ,
…,   z,   … ,',   s,   ›, œ, …,   z,    ¨Y,   …, £, …, ß, …,  ¨y,   …  ", ", …,

If you need to store and sort characters outside of the ASCII character set, see Chapter 10, in which the setting of the character set at launch of MySQL is discussed.

Caution

If you are using the monitor mysql.exe under Windows, it runs in a DOS-like environment, which has its own character set, which is not compatible with that of MySQL. For this reason, SQL commands with special characters seem to be executed incorrectly in mysql.exe. In reality, MySQL is working correctly, but because of the DOS character set—which is valid for mysql.exe—incorrect character codes for such special characters as åäê are transmitted to the MySQL server.

Pattern Matching with LIKE

MySQL offers two operators for pattern matching. The simpler, and ANSI-compatible, of these is LIKE. As with normal character string comparison, there is no case distinction. In addition, there are two wild cards:

_ | an arbitrary character
% | arbitrarily many (even 0) characters (but not for NULL)
\_ | the character _
\% | the character %
SELECT 'MySQL' LIKE '%sql'
1

Pattern Matching with REGEXP

Considerably wider scope in the formulation of a pattern is offered by REGEXP and the equivalent command RLIKE. The relatively complicated syntax for the pattern corresponds to the Unix commands grep and sed.

REGEXP Search Patterns

 | Pattern Definition
abc | character string abc
(abc) | character string abc (made into a group)
[abc] | one of the characters a, b, c
[a-z] | one of the characters a–z
[^abc] | none of these characters (any others)
. | any character
 | Pattern Appearance
x | expression x must appear once
x | y | expression x or y must appear once
x? | expression x may appear at most once
x* | expression x may appear arbitrarily often (including not at all)
x+ | expression x may appear arbitrarily often (but at least once)
x{n} | expression x must appear exactly n times
x{,n} | expression x may appear at most n times
x{n,} | expression x must appear at least n times
x{n,m} | expression x must appear at least n and atmost m times
^ | wild card for the beginning of a character string
$ | wild card for the end of a character string
\x | special character x (e.g., \$ for $)

As with LIKE, there is no case distinction. Please note that REGEXP is successful when the search pattern is found somewhere within the character string. The search pattern is thus not required to describe the entire character string, but only a part of it. If you wish to encompass the entire character string, then you must use ^ and $ in the search pattern.

Pointer

The above table contains only the most important elements of REGEXP patterns. A complete description can be obtained under Unix/Linux with man 7 regex. This can also be found on the Internet, for example, at http://linux.ctyme.com/man/alpha7.htm

Binary Character String Comparison

Character strings are normally compared without case being taken into consideration. Thus 'a' = 'A' returns 1 (true). If you wish to execute a binary comparison, then you must place BINARY in front of one of the operands. BINARY is a cast operator; that is, it alters the data type of one of the operands (in this case it changes a number or character string into a binary object). BINARY can be used both for ordinary character string comparison and for pattern matching with LIKE and REGEXP:

SELECT 'a'='A', BINARY 'a' = 'A', 'a' = BINARY 'A'
    1, 0, 0

Logical Operators

Logical operators likewise return 0 or 1, or NULL if one of the operands is NULL. This holds also for NOT; that is, NOT NULL again returns NULL.


Variables and Constants

MySQL supports two types of variables: user variables (@name) and system variables @@name. Furthermore, there are many special cases that arise from the MySQL syntax. Note that MySQL is case-sensitive in the naming of variables.

User Variables (@name)

As the following examples demonstrate, you can assign your own variables in SQL commands:

SET @varname = 3
SELECT @varname := 3
SELECT @varname = MIN(birthdate) FROM table

These variables can be used in other SQL commands. Their contents disappear at the end of the connection.

System Variables (@@name)

Various pieces of status information are accessible over the MySQL server as variables. There are two ways of looking at these variables: SHOW VARIABLES and SELECT.

SHOW VARIABLES can display an entire list of variables:

SHOW VARIABLES LIKE 'b%'
 Variable_name           Value
--------------------------------------
 back_log                50
 basedir                 C:\mysql\
 binlog_cache_size       32768
 bulk_insert_buffer_size 8388608
Remark

Please note that for some strange reason, SHOW ignores a number of variables (e.g., autocommit). For such variables, you must use SELECT.

With SELECT, the variable name must be given exactly, prefixed by two @ signs:

SELECT @@binlog_cache_size
 @@binlog_cache_size
----------------------
 32768

Global System Variables Versus System Variables at the Connection Level

In its system variables, MySQL distinguishes between SESSION and GLOBAL variables. SESSION variables are valid only for the current session (connection), while GLOBAL variables hold for the entire server:

SELECT @@wait_timeout            -- Session (connection level)
SELECT @@session.wait_timeout    -- Session (connection level)
SELECT @@global.wait_timeout     -- Global

System variables can also be changed. According to whether the change is only for the current connection or should be valid globally, the following syntax variants are available. Note that with SET, you may omit the two @ symbols. One cannot change system variables with SELECT:

SET @@wait_timeout = 10000           -- Session (connection level)
SET @@session.wait_timeout = 10000   -- Session (connection level)
SET SESSION @@wait_timeout = 10000   -- Session (connection level)
SET @@global.wait_timeout = 10000    -- Global
SET GLOBAL @@wait_timeout = 10000    -- Global

Variables at the global level can be changed by users possessing the Super privilege. When a global variable is changed, the new value holds for all new connections, but not for those already in existence.

Changes in SESSION variables, on the other hand, hold only until the end of the current connection. When a new connection is made, the global default value again holds.

Pointer

This book does not contain a complete description of all MySQL system variables. In the description of the command SET later in this chapter, only the most important variables are described.

A complete list of GLOBAL and SESSION variables can be found in the MySQL documentation:

http://www.mysql.com/doc/en/System_Variables.html

You may end up at the key word LOCAL, which has the same meaning in this context as SESSION.

Enlightenment on the contents of the variables can be found in the MySQL documentation at the following pages:

http://www.mysql.com/doc/en/Server_parameters.html
http://www.mysql.com/doc/en/SET_OPTION.html
http://www.mysql.com/doc/en/SHOW_VARIABLES.html

Special Cases (Options, Passwords)

The command SET can also be used to change some of the MySQL options and the connection passwords. These options look like variables, but they cannot be read with SELECT or SHOW VARIABLES. The following list names the most frequently used special forms. A complete list can be found later in the chapter, when the syntax of SET is elucidated:

SET AUTOCOMMIT = 0 | 1
SET LAST_INSERT_ID = #
SET PASSWORD = PASSWORD('xxx')
SET PASSWORD FOR user@hostname = PASSWORD('xxx')
SET QUERY_CACHE_TYPE = 0 | 1 | 2
SET QUERY_CACHE_TYPE = OFF | ON | DEMAND
SET SQL_SAFE_UPDATES = 0 | 1
SET SQL_SELECT_LIMIT = value | DEFAULT
SET TIMESTAMP = timestamp_value | DEFAULT

Constants

Starting with version 4.1, MySQL recognizes the constants TRUE (1) and FALSE (0).


MySQL Data Types

 | Integers
TINYINT(m) | 8-bit integer (1 byte); the optional value m specifies the desired column width in SELECT results (maximum display width), and has no influence on the allowable range of numbers
SMALLINT(m) | 16-bit integer (2 bytes)
MEDIUMINT(m) | 24-bit integer (3 bytes)
INT(m), INTEGER(m) | 32-bit integer (4 bytes)
BIGINT(m) | 64-bit integer (8 bytes)
 | Floating-Point Number
FLOAT(m, d) | floating-point number, 8-place precision (4 bytes); the optional values m and d specify the desired number of places before and after the decimal point in SELECT results; the values have no influence over the way the number is stored
DOUBLE(m, d) | floating-point number, 16-place precision (8 bytes)
REAL(m, d) | synonym for DOUBLE
DECIMAL(p, s) | fixed-point number, stored as string; arbitrary number of places (1 byte per digit + 2 bytes overhead); p specifies the entire number of places, where s is the number of places after the decimal point; default is DECIMAL(10,0)
NUMERIC, DEC | synonyms for DECIMAL
 | Date, Time
DATE | date in the form '2003-12-31',range 1000-01-01 to 9999-12-31 (3 bytes)
TIME DATETIME | time in the form '23:59:59',range ±838:59:59 (3 bytes) combination of DATE and TIME in the form '2003-12-31 23:59:59' (8 bytes)
YEAR | year 1900–2155 (1 byte)
TIMESTAMP(m) | date and time in the form 20031231235959 for times between 1970 and 2038 (4 bytes); the optional value m specifies the number of places in SELECT results; m=8,for example, has the effect that only year, month, and day are displayed
 | Character Strings (ANSI)
CHAR(n) | string with preset length, max 255 characters (nbytes)
VARCHAR(n) | string with variable length, max nbytes (n<256)memory requirement for latin1 characters: 1 byte per character (actual length) + 1
NCHAR(n) | Unicode character string (corresponds to CHAR(n) CHARSET utf8)
NCHAR VARCHAR(n) | Unicode character string with variable length (corresponds to VARCHAR(n) CHARSET utf8)
TINYTEXT | string with variable length, max 255 bytes
TEXT MEDIUMTEXT LONGTEXT | string with variable length, max 216 − 1 bytes string with variable length, max 224 − 1 bytes string with variable length, max 232 − 1 bytes
 | Binary Data
TINYBLOB | binary data with variable length, max 255 bytes
BLOB MEDIUMBLOB LONGBLOB | binary data with variable length, max 216 − 1 bytes binary data with variable length, max 224 − 1 bytes binary data with variable length, max 232 − 1 bytes
 | Geometric Data (since MySQL 4.1)
GEOMETRY | a general geometric object
 | Other
ENUM | enumeration of max 65535 character strings (1 or 2 bytes)
SET | enumeration of max 255 character strings (1–8 bytes)

Options for MySQL Data Types

In the definition of columns (CREATE TABLE, ALTER TABLE), each column can be assigned different options. The following table summarizes these options. Note that not all options are applicable to all data types:

NULL | specifies that the column may contain the value NULL;this setting holds by default
NOT NULL | forbids the value NULL
DEFAULT 'xxx' | specifies the default value xxx to be used if no other input value is specified
PRIMARY KEY | defines the primary key column
AUTO_INCREMENT | results in an automatically increasing number being inserted in the column; it can be used for only one column, with integer values; moreover, the options NOT NULL and PRIMARY KEY must be specified (instead of PRIMARY KEY,the column can be given a UNIQUE index)
UNSIGNED | integers are stored without a sign; note that calculations are also made without signs
ZEROFILL | integers in SELECT results are padded with zeros (e.g., five-digit numbers 00123, 01234)
BINARY | with CHAR and VARCHAR columns, comparison and sort operations are executed in binary; uppercase letters are ordered before lowercase; this is more efficient, but generally impractical when sorted results are to be displayed

Starting with MySQL 4.1, the character set and sort order can be set for text columns. This is the syntax: [DEFAULT] CHARACTER SET csname [COLLATE collname].


Command Overview (Thematic)

In the following section, SQL commands will be listed in alphabetical order. As a supplementary aid to orientation, we provide here a systematic overview:

Database Queries, Data Manipulation

SELECT | queries existing data records (data search)
INSERT | inserts a new record
REPLACE | replaces an existing record
UPDATE | changes an existing record
DELETE | deletes selected records
TRUNCATE TABLE | deletes all records
LOAD DATA | inserts records from a text file
HANDLER | reads records more efficiently than SELECT (since MySQL 4.0)

Transactions (Only for Some Table Types)

BEGIN | begins a group of SQL commands
COMMIT | confirms all executed commands
ROLLBACK | cancels all executed commands

Generate Databases/Tables, Change Database Design

ALTER DATABASE ALTER TABLE CREATE DATABASE | makes changes to the database (since MySQL 4.1) changes individual columns of a table, adds indexes, etc. creates a new database
CREATE FUNCTION CREATE INDEX | includes a function programmed in C/C++ into MySQL creates a new index for a table
CREATE TABLE | creates a new table
DROP DATABASE | deletes an entire database
DROP INDEX | deletes an index
DROP TABLE | deletes an entire table
RENAME TABLE | renames a table

Table Management (General)

FLUSH TABLES | closes all table files and then reopens them
LOCK TABLE | blocks tables for (write) access by other users
UNLOCK TABLES | releases a table that has been locked with LOCK

Management of MyISAM Tables

ANALYZE TABLE | returns information about internal index management
BACKUP TABLE | copies table files into a backup directory
CHECK TABLE | tests the table file for consistency
OPTIMIZE TABLE | optimizes memory usage in tables
REPAIR TABLE | attempts to repair errors in table files
RESTORE TABLE | restores a table that has been backed up with BACKUP

Information on the Database Structure, Other Management Information

DESCRIBE | corresponds to SHOW COLUMNS
EXPLAIN | explains how a SELECT command is executed internally
SHOW | gives information on databases, tables, fields, etc.

Administration, Access Privileges, etc.

FLUSH | empties MySQL intermediate storage or inputs it anew
GRANT | grants additional access privileges
KILL | terminates a process
REVOKE | restricts access privileges
RESET QUERY CACHE | deletes the query cache
SET | changes the contents of MySQL system variables
SHOW | shows the MySQL status, system variables, processes, etc.
USE | changes the active database

Replication (Master)

PURGE MASTER LOGS | deletes old logging files
RESET MASTER | deletes all logging files
SET SQL_LOG_BIN=0/1 | deactivates/activates binary logging
SHOW BINLOG EVENTS | returns a list of entries from the active logging file (since MySQL 4.0)
SHOW MASTER LOGS | returns a list of all binary files
SHOW MASTER STATUS | gives the current logging file
SHOW SLAVE HOSTS | returns a list of all registered slaves (from MySQL 4.0)

Replication (Slave)

CHANGE MASTER TO | changes replication settings in master.info
LOAD DATA FROM | copies all tables from master to slave (since MySQL 4.0)
LOAD TABLE FROM | copies a table from master to slave
RESET SLAVE | reinitializes master.info
SHOW SLAVE STATUS | displays the contents of master.info
SLAVE START/STOP | starts or ends replication


Command Reference (Alphabetical)

In the following reference section, the following syntax is in force:

  • Optional parts of a command are written in square brackets ([option]).

  • Alternatives are separated by a vertical bar (variant1 | variant2 | variant3).

Start example

ALTER DATABASE dbname actions

Since MySQL 4.1, with ALTER DATABASE you can change global database attributes. The settings are stored in the file dbname/db.opt.

  • actions: In MySQL 4.1 alpha, only an action command has been implemented. CHARACTER SET charset [COLLATE collname] specifies which character set the database should use by default and what sort order should be used with it. Without this setting, the database will use the default character set of the MySQL server. Moreover, with CREATE TABLE, a different character set can be chosen for the table or for individual columns.

End example
Start example

ALTER TABLE tblname tbloptions

ALTER TABLE can be used to change various details of the structure of a table. In the following, we present an overview of the syntactic variants.

In the syntactically simplest form that we shall show here, ALTER TABLE changes the table options. The possible options are described in CREATE TABLE. The command can be used, for example, to change the type of a table (e.g., from MyISAM to InnoDB).

Note that with many ALTER TABLE variants, the table must be re-created. To do this, MySQL creates a new table X with the new table properties, and then copies all the records into this new table. Then the existing table is renamed Y, and table X is renamed tblname. Finally, Y is deleted. On large tables, this can take considerable time and temporarily use a great deal of hard-disk space.

End example
Start example

ALTER TABLE tblname ADD newcolname coltype coloptions [FIRST | AFTER existingcolumn]

This command adds a new column to a table. The definition of the new column takes place as with CREATE TABLE. If the position of the new column is not specified with FIRST or AFTER,then the new column will be the last column of the table.

The following example adds a new column ts with data type TIMESTAMP to the authors table:

ALTER TABLE authors ADD ts TIMESTAMP
End example
Start example

ALTER TABLE tblname ADD INDEX [indexname] (indexcols ... ) ALTER TABLE tblname ADD FULLTEXT [indexname] (indexcols ... ) ALTER [IGNORE] TABLE tblname ADD UNIQUE [indexname] (indexcols ... ) ALTER [IGNORE] TABLE tblname ADD PRIMARY KEY (indexcols ... ) ALTER TABLE tblname ADD SPATIAL INDEX (indexcol)

These commands create a new index for a table. If no indexname is specified, then MySQL simply uses the name of the indexed column.

The optional key word IGNORE comes into play if several identical fields are discovered in the creation of a UNIQUE or primary index. Without IGNORE, the command will be terminated with an error, and the index will not be generated. With IGNORE, such duplicate records are simply deleted.

A spatial index for geometric data can be created starting with MySQL 4.1. The column indexcol must have data type GEOMETRY and the attribute NOT NULL.

End example
Start example

ALTER TABLE tblname ADD FOREIGN KEY [name] (column1) REFERENCES table2 (column2) [ON DELETE {CASCADE | SET NULL | NO ACTION | RESTRICT}] [ON UPDATE {CASCADE | SET NULL | NO ACTION | RESTRICT}]

This command defines a foreign key constraint. This means that the foreign key tblname.column1 refers to table2.column2, and the table driver should ensure that no references point to nowhere.

The optional ON DELETE and ON UPDATE clauses specify how the table driver is to react to damage to integrity on DELETE and UPDATE commands (see Chapter 8 for details). By default, the condition is STRICT, meaning that potential damage to integrity results in the command not being issued and an error message being triggered.

Currently (MySQL 4.0.n), foreign key constraints can be applied only to InnoDB tables. Both column1 and column2 must be given indexes and must be of the same data type.

End example
Start example

ALTER TABLE tblname ALTER colname SET DEFAULT value ALTER TABLE tblname ALTER colname DROP DEFAULT

This command changes the default value for a column or table or deletes an existing default value.

End example
Start example

ALTER TABLE tblname CHANGE oldcolname newcolname coltype coloptions

This command changes the default value for a column in a table or deletes an existing default value. The description of the column proceeds as with CREATE TABLE, which you may refer to. If the column name is to remain unchanged, then it must be given twice (that is, oldcolname and newcolname are identical). Even if ALTER TABLE is used only to change the name of a column, both coltype and coloptions must be completely specified.

End example
Start example

ALTER TABLE tblname DISABLE KEYS ALTER TABLE tblname ENABLE KEYS

Since MySQL 4.0, ALTER TABLE ... DISABLE KEYS has the effect that all nonunique indexes are no longer automatically updated with INSERT, UPDATE, and DELETE commands. ALTER TABLE ... ENABLE KEYS restores activation and updating of indexes.

The two commands should be used for carrying out extensive revisions to tables in the most efficient manner possible. (The reconstruction of indexes with ENABLE KEYS costs considerably less time than the constant updating with each altered record.)

End example
Start example

ALTER TABLE tblname DROP colname ALTER TABLE tblname DROP INDEX indexname ALTER TABLE tblname DROP PRIMARY KEY ALTER TABLE tblname DROP FOREIGN KEY foreign_key_id

The first three commands delete a column, an index, or the primary index. The fourth command, since MySQL 4.0.13, deletes the specified foreign key constraint. You must first determine with SHOW CREATE TABLE the foreign_key_id of the index to be deleted.

End example
Start example

ALTER TABLE tblname MODIFY colname coltype coloptions

This command functions like ALTER TABLE ... CHANGE (see above). The only difference is that the column cannot be changed, and thus the name needs to be given only once.

End example
Start example

ALTER TABLE tblname RENAME AS newtblname

This command renames a table. See also RENAME TABLE.

End example
Start example

ALTER TABLE tblname TYPE typename

This command changes the type of the table (the table driver). Allowable table types are InnoDB and MyISAM. Note that a type change is possible only if the new table driver supports all the properties of the table. For example, the InnoDB table driver currently does not support a full-text index. If you wish to change a MyISAM table to InnoDB, you must first remove any full-text index (ALTER TABLE tblname DROP indexname).

End example
Start example

ANALYZE TABLE tablename1, tablename2, ...

ANALYZE TABLE performs an analysis of the indexed values of a column. The results are stored, and in the future, this speeds up index access to data records a bit.

ANALYZE TABLE can be used only with tables of type MyISAM. Instead of this command, the external program myisamchk -a tblfile can be used.

End example
Start example

BACKUP TABLE tblname TO '/backup/directory'

BACKUP TABLE copies the files for the specified MyISAM table into a backup directory. The table can be re-created with RESTORE TABLE.

Under Unix/Linux, the backup directory for the account under which MySQL is executed must be writable. BACKUP and RESTORE do not work for InnoDB tables.

End example
Start example

BEGIN

If you are working with transaction-capable tables, then BEGIN introduces a new transaction. The following SQL commands can then be confirmed with COMMIT or revoked with ROLLBACK. (All changes to tables are executed only via COMMIT.)

Further information and examples on the topic of transactions can be found in Chapter 8.

Since MySQL 4.0.11, you can use the ANSI-conforming command START TRANSACTION instead of BEGIN.

End example
Start example

CHANGE MASTER TO variable1=value1, variable2=value2, ...

With this command, the replication settings for slave are carried out. The settings are stored in the file master.info. The command can be used only for slave computers in a replication system, and it requires the Super privilege. It recognizes the following variable names:

MASTER_HOST | specifies the hostname or IP number of the master computer
MASTER_USER | specifies the username used for communication with the master computer
MASTER_PASSWORD | specifies the associated password
MASTER_PORT | specifies the port number of the master computer (normally 3306)
MASTER_LOG_FILE | specifies the current logging file on the master computer
MASTER_LOG_POS | specifies the current read position within the logging file on the master computer
End example
Start example

CHECK TABLE tablename1, tablename2 ... [TYPE=QUICK ]

CHECK TABLE tests the internal integrity of the database file for the specified table. Any errors that are discovered are not corrected.

With MyISAM tables, instead of this command, the external program myisamchk -m tblfile can be used.

End example
Start example

COMMIT

COMMIT ends a transaction and stores all changes in the database. (Instead of executing COMMIT, you can cancel the pending changes with ROLLBACK.) BEGIN/COMMIT/ROLLBACK function only if you are working with transaction-capable tables. Further information and examples on transactions can be found in Chapter 8.

End example
Start example

CREATE DATABASE dbname [options]

CREATE DATABASE generates the specified database. (More precisely, an empty directory is created in which tables belonging to the new database can be stored.) Note that database names are case-sensitive. This command can be executed only if the user has sufficient access privileges to create new databases.

  • options: Since MySQL 4.1, you can specify the default character set for a table: [DEFAULT] CHARACTER SET charset [COLLATE collname]. The optional key word DEFAULT has no function (that is, it does not matter whether you specify it or not). With COLLATE you can select the sort order if there is more than one for the character set in question. If you do not specify the CHARACTER SET, then the default character set of the server is used.

End example
Start example

CREATE FUNCTION fnname RETURNS datatype SONAME libraryname

CREATE FUNCTION makes it possible to bring a function in an external library into MySQL. Such functions are called user-defined functions,or UDFs for short.

If you believe that this means that you can simply equip MySQL with a variety of new functions, you are sadly mistaken: The syntax of CREATE FUNCTION is simple, but the problem is that first the code for the function must be written in C or C++ and then compiled into a library. This requires a degree of background knowledge about how functions work in MySQL (and, of course, the requisite tools, like compilers). Further information on creating MySQL functions can be found in the MySQL documentation:

http://www.mysql.com/doc/en/Extending_MySQL.html

It is planned for MySQL 5.0 that in addition to UDFs, there will be the possibility of realizing custom SQL functions and commands in the form of stored procedures (SPs). Stored procedures are simpler to program, but not as efficient as UDFs.

End example
Start example

CREATE [UNIQUE|FULLTEXT] INDEX indexname ON tablename (indexcols ... )

CREATE INDEX enlarges an existing database to include an index. As indexname, the name of the column is generally used. CREATE INDEX is not a freestanding command, but merely an alternative form of ALTER TABLE ADD INDEX/UNIQUE, which you should see for details.

End example
Start example

CREATE [TEMPORARY] TABLE [IF NOT EXISTS] tblname
( colname1 coltype coloptions reference, colname2
coltype coloptions reference ... [, index1, index2 ... ]
) [tbloptions]

CREATE TABLE generates a new table in the current database. If a database other than the current one is to be used, the table name can be specified in the form dbname.tblname. If the table already exists, an error message results. There is no error message if IF NOT EXISTS is used, but in this case, the existing table is not affected, and no new table is created.

If the key word TEMPORARY is used (as of version 3.23), then the table that is created is a temporary one. If a temporary table is created and a like-named, but nontemporary, table already exists, the temporary table is created without an error message. The old table is preserved, but it is masked by the temporary table. If you want your temporary table to exist only in RAM (for increased speed), you must also specify TYPE= HEAP.

The creation of regular tables requires the Create privilege. Since version 4.0, the creation of temporary tables requires the Create Temporary Table privilege.

  • colname: Name of the column.

  • coltype: Data type of the column. A list of all MySQL data types (INT, TEXT, etc.) appears in Chapter 18.

  • coloptions: Here, several attributes (options) can be specified:

    • NOT NULL | NULL: NOT NULL indicates that no NULL values may be stored in the column, and thus the column may not have empty cells. (By default, columns are created with the attribute NULL.) ◦

    • DEFAULT defaultval: DEFAULT specifies a default value that is then stored if no value is specified. ◦

    • AUTO_INCREMENT | IDENTITY: AUTO_INCREMENT indicates that MySQL inserts a unique numerical value for each new record (ideal for primary key fields). ◦

    • PRIMARY KEY: PRIMARY KEY signifies that the field is to be used as a primary key. (MySQL creates a UNIQUE index for the field. The field must have the attribute NOT NULL.) ◦

    • CHARACTER SET charset [COLLATE collname]: Since MySQL 4.1, with CHARACTER SET, you can specify a separate character set for each table column. COLLATE determines the sort order if for the given character set there is more than one from which to choose. ◦

    • COMMENT: With COMMENT you can also, since MySQL 4.1, save a brief comment about the meaning of the column.

  • reference: MySQL provides various key words for the declaration of foreign keys in keeping track of referential integrity, e.g., REFERENCES tblname (idcolumn). These key words are currently ignored, however (and with no error message). Even if you use InnoDB tables, the foreign key constraints for such tables must be specified within the confines of the index definition.

  • index: KEY or INDEX defines a usual index spanning one or more columns. UNIQUE defines a unique index (that is, in the column or columns, no identical values or groups of values can be stored). With both variants an arbitrary index name may be given for the internal management of the index. PRIMARY KEY likewise defines a UNIQUE index. Here, however, the index name is predefined: It is, not surprisingly, PRIMARY. With FULLTEXT, an index is declared for full-text search (in MySQL 4.0, only with MyISAM tables). Since MySQL 4.1, an index for GEOMETRY data can be created with SPATIAL INDEX; indexcol must be defined with the attribute NOT NULL.

    • KEY | INDEX [indexname] (indexcols ... )

    • UNIQUE [INDEX] [indexname] (indexcols ... )

    • PRIMARY KEY (indexcols ... )

    • FULLTEXT [indexname] (indexcols ... )

    • SPATIAL INDEX [indexname] (indexcol)

    Foreign key constraints: If youare using InnoDB tables, here you can formulate foreign key constraints. The syntax is as follows:

    FOREIGN KEY [name] (column1) REFERENCES table2 (column2) [ON DELETE {CASCADE | SET NULL | NO ACTION | RESTRICT}] [ON UPDATE {CASCADE | SET NULL | NO ACTION | RESTRICT}]

    This means that tblname.column1 is a foreign key that refers to table2.column2. The table driver ensures that no references can point to nowhere (to nonexistent values in table2.column2).

    With the optional ON DELETE and ON UPDATE clauses, you can specify how the table driver should react to damage to integrity on UPDATE and DELETE commands (for details, see Chapter 8). By default, RESTRICT is in force; that is, if integrity is to be damaged, the command in question is not executed, and an error is triggered.

  • tbloptions: Here various table options can be specified, though here we shall exhibit only the most important of them. Not all options are possible with every table type. Information on the different table types and their variants (MyISAM static, dynamic, or compressed) can be found in Chapter 5. InnoDB tables were described in Chapter 8.

    • TYPE= ISAM | MYISAM | HEAP | BDB | MERGE | INNODB. ◦

    • ROW_FORMAT= default | dynamic | static | compressed. ◦

    • AUTO_INCREMENT = n: AUTO_INCREMENT gives the initial value for the counter of an AUTO_INCREMENT column (e.g., 100000 if you wish to have six-digit integers). ◦

    • CHECKSUM = 0 | 1: CHECK_SUM=1 has the effect that a check sum is stored for each data record, which helps in reconstruction if the database is damaged. ◦

    • PACK_KEYS = 0 | 1: PACK_KEYS=1 results in a smaller index file. This speeds up read access, but slows down changes. ◦

    • DELAY_KEY_WRITE=0|1:DELAY_KEY_WRITE = 1 results in indexes not being updated each time a change to a record is made. Rather, they are updated every now and then. ◦

    • With COMMENT you can save a brief text, for example, to describe the purpose of the table. The comment can be read with SHOW CREATE DATABASE dbname: COMMENT= 'comment'. ◦

    • Since MySQL 4.1, you can specify the character set and sort order for a table. (You can also set these parameters for a single column.) If no character set is specified, then the default character set for the table or that of the MySQL server is used: [DEFAULT] CHARACTER SET charset [COLLATE collname].

The CREATE TABLE syntax contains some duplication. For example, a primary index can be declared in two different ways, either as an attribute of a column (coloptions) or as an independent index (index). The result is, of course, the same. It is up to you to decide which form you prefer. Here is an example:

CREATE TABLE test   (id INT NOT NULL AUTO_INCREMENT,
                     data INT NOT NULL,
                     txt VARCHAR(60),
                     PRIMARY KEY (id))

Additional examples can be found in Chapter 5, under the topic database design.

End example
Start example

CREATE [TEMPORARY] TABLE [IF NOT EXISTS] tblname
[( newcolname1 coltype coloptions reference,
newcolname2 coltype coloptions reference ...
[, key1, key2 ... ]

)]
[tbloptions]
[IGNORE | REPLACE]SELECT ...

With this variant of the CREATE TABLE command, a table is filled with the result of a SELECT command. The individual columns of the new table take their types from the data types of the SELECT command and thus do not have to be (and may not be!) declared explicitly.

Unfortunately, neither indexes nor attributes such as AUTO_INCREMENT are carried over from the old table. There can also be changes in column types, such as VARCHAR columns turning into CHAR.

If an index is to be created in the new table for individual columns (e.g., PRIMARY KEY (id)), then this can be specified. Moreover, there is the option of defining new columns (e.g., an AUTO INCREMENT column). On the other hand, if you want an AUTO INCREMENT in the old table to become again an AUTO INCREMENT column in the new table, then you must execute an ALTER TABLE CHANGE command after the CREATE command.

The key words IGNORE and REPLACE specify how MySQL should behave if several records with the same value are placed by the command into a UNIQUE column. With IGNORE, the existing record is retained, and new records are ignored. With REPLACE existing records are replaced by the new ones. If neither option is used, an error message results.

End example

Examples

This command provides a simple mechanism in MySQL for copying complete tables:

CREATE TABLE backuptable SELECT * FROM table

However, you may also copy only parts of a table into a temporary table:

CREATE TEMPORARY TABLE tmp
  SELECT id, authname FROM authors WHERE id<20

If the specification of the new table is to meet that of the old table exactly, then frequently, a great deal of cleanup work is necessary, as indicated by the following commands:

CREATE TABLE table2 SELECT * FROM table1
ALTER TABLE table2 ADD PRIMARY KEY(id)
ALTER TABLE table2 CHANGE titleID titleID INT AUTO_INCREMENT
ALTER TABLE table2 CHANGE title title VARCHAR(100)

If a table is to be copied exactly, it is better to create a new table with a CREATE TABLE command and then copy the data with INSERT INTO table2 SELECT * FROM table1. Since MySQL 4.1 there is the convenient command CREATE TABLE table2 LIKE table1.

Start example

CREATE [TEMPORARY] TABLE [IF NOT EXISTS] newtable LIKE oldtable

This command creates a new, empty, table newtable corresponding to the declaration of the existing table oldtable. This CREATE TABLE variant is available since version 4.1.

End example
Start example

DELETE [deleteoptions] FROM tablename [WHERE condition] [ORDER BY ordercolumn [DESC]] [LIMIT maxrecords ]

DELETE deletes the records in a table encompassed by condition.

  • deleteoptions: The LOW_PRIORITY option has the effect that the data records are deleted only when all read operations are complete. (The goal of this option is to avoid having SELECT queries unnecessarily delayed due to DELETE operations.)

    The option QUICK has the effect that during deletion, an existing index is not optimized. This speeds up the DELETE command, but it can lead to a somewhat inefficient index.

  • condition: This condition specifies which records are to be deleted. For the syntax of condition,see SELECT.

  • ordercolumn: With ORDER BY you can first sort the data to be deleted. This makes sense only in combination with LIMIT, in order, for example, to delete the first or last ten records (according to some sort criterion).

  • maxrecords: With LIMIT, the maximum number of records that may be deleted is specified.

If DELETE is executed without conditions, then all records of the table are deleted (so be careful!). DELETE without conditions cannot be part of a transaction. If a transaction is open, it is closed with COMMIT before the DELETE command is executed. If you wish to delete large tables completely, it is more efficient to use the command TRUNCATE.

End example
Start example

DELETE [deleteoptions] table1, table2, ...
FROM table1, table2, table3, ...
[USING columns]
WHERE conditions

This variant of DELETE (available since version 4.0) deletes records from tables table1, table2, etc., where the data of additional tables (table3, etc.) are considered in the search criteria.

After DELETE, all tables from which data are to be deleted must be specified. After FROM, all DELETE tables must appear, as well as any additional tables that serve only in formulating the search criteria.

  • deleteoptions: Here you can specify options as in a usual DELETE command.

  • columns: Here fields that link the tables can be specified (see also SELECT). This assumes that the linking field has the same name in both tables.

  • conditions: In addition to the usual delete criteria, here one may specify linking conditions (e.g., WHERE table1.id = table2.forgeinID).

End example
Start example

DESCRIBE tablename [ columnname ]

DESCRIBE returns information about the specified table in the current database (or about a particular column of this table). Instead of columnname, a pattern with the wild cards _ and % can be given. In this case, DESCRIBE displays information about those columns matching the pattern. DESCRIBE returns the same information as EXPLAIN or SHOW TABLE or SHOW COLUMN.

End example
Start example

DO selectcommand

DO is a variant of SELECT and has basically the same syntax. The difference between the two is that DO returns no results. For example, DO can be used for variable assignment, for which it is somewhat faster than SELECT (thus, for example, DO @var:=3).

End example
Start example

DROP DATABASE [IF EXISTS] dbname

DROP DATABASE deletes an existing database with all of its data. This cannot be undone, so be careful! If the database does not exist, then an error is reported. This error can be avoided with an IF EXISTS.

In the execution of this command, all files in the directory dbname with the following endings are deleted, among others: .BAK, .DAT, .HSH, .ISD, .ISM, .MRG, .MYD, .MYI, .db, .frm.

End example
Start example

DROP FUNCTION fnname

DROP FUNCTION deactivates an auxiliary function that was made available to MySQL earlier with CREATE FUNCTION.

End example
Start example

DROP INDEX indexname ON tablename

DROP INDEX removes an index from the specified table. Usually, indexname is the name of the indexed column, or else PRIMARY for the primary index.

End example
Start example

DROP TABLE [IF EXISTS] tablename1, tablename2, ...

DROP TABLE deletes the specified tables irrevocably. The option IF EXISTS avoids an error message if the tables do not exist.

Note that until MySQL 4.0, DROP TABLE automatically ends a running transaction (COMMIT). This holds even in the case of a temporary table. Since MySQL 4.1 the behavior has changed somewhat: DROP TABLE ends a transaction only for regular tables, not for temporary ones.

End example
Start example

EXPLAIN tablename

EXPLAIN returns a table with information about all the columns of a table (field name, field type, index, default value, etc.). The same information can be determined as well with SHOW COLUMNS or DESCRIBE,orvia an external program such as mysqlshow.

End example
Start example

EXPLAIN SELECT selectcommand

EXPLAIN SELECT returns a table with information about how the specified SELECT command was executed. These data can help in speed optimization of queries, and in particular in deciding which columns of a table should be indexed. (The syntax of selectcommand was described under SELECT. An example of the use of EXPLAIN SELECT and a brief description of the resulting table can be found in Chapter 5.)

End example
Start example

FLUSH flushoptions

FLUSH empties the MySQL internal intermediate storage. Any information not stored already is thereby stored in the database. The execution of FLUSH requires the RELOAD privilege.

  • flushoptions: Here one may specify which cache(s) should be emptied. Multiple options should be separated by commas.

    • DES_KEY_FILE: Reloads the key files for the functions DES_ENCRYPT and DES_DECRYPT. ◦

    • HOSTS: Empties the host cache table. This is necessary especially if in the local network the arrangement of IP numbers has changed.

    • LOGS: Closes all logging files and then reopens them. In the case of update logs, a new logging file is created, whereby the number of the file ending is increased by 1 (file.003 ® file.004).

    • QUERY CACHE: Defragments the query cache so that it can use its memory more efficiently. The cache is not emptied. ◦

    • PRIVILEGES: Reloads the privileges database mysql (corresponds to mysqladmin reload).

    • STATUS: Sets most status variables to 0.

    • TABLES: Closes all open tables.

    • TABLE[S] tblname1, tblname2, ... : Closes the specified tables.

    • TABLES WITH READ LOCK: As above, except that additionally, LOCK is executed for all tables, which remains in force until the advent of a corresponding UNLOCK table.

    • USER_RESOURCES: Resets the counters for MAX_QUERIES_PER_HOUR, MAX_UPDATES_PER_HOUR, and MAX_CONNECTIONS_PER_HOUR (see maxlimits under GRANT).

Most FLUSH operations can also be executed through the auxiliary program mysqladmin.

End example
Start example

GRANT privileges ON objects TO users [IDENTIFIED BY 'password'] [REQUIRE ssloptions ] [WITH GRANT OPTION | maxlimits ]

GRANT helps in the allocation of access privileges to database objects.

  • privileges: Several privileges may be specified (separated by commas):

    ALTER, CREATE, DELETE, DROP, FILE, INDEX, PROCESS,
    REFERENCES, RELOAD, SELECT, SHUTDOWN, UPDATE.

    If you wish to set all (or no) privileges, then specify ALL (or USAGE). (The second variant is useful if you wish to create a new MySQL user to whom as of yet no privileges have been granted.) The Grant privilege can be set only via WITH GRANT OPTION; that is, ALL does not include the Grant privilege.

    If the privileges are to hold only for certain columns of a table, then specify the columns in parentheses. For example, you may specify GRANT SELECT(columnA, columnB).

  • objects: Here databases and tables are specified. The following syntactic variants are available:

    databasename.tablename

    only this table in this database

    databasename.*

    all tables in this database

    tablename

    only this table in the current database

    *

    all tables of the current database

    *.*

    global privileges

    Wild cards may not be used in the database names.

  • users: Here one or more (comma-separated) users may be specified. If these users are not yet known to the user table, they are created. The following variants are allowed:

    username@hostname

    only this user at hostname

    'username'@'hostname'

    as above, with wild cards

    username

    this user on all computers

    ''@hostname

    all users on hostname

    ''

    all users on all computers

  • password: Optionally, with IDENTIFIED BY, a password in plain text can be specified. GRANT encrypts this password with the function PASSWORD before it is entered in the user table. If more than one user is specified, then more than one password may be given:

    TO user1 IDENTIFIED BY 'pw1', user2 IDENTIFIED BY 'pw2', ...

  • ssloptions: If access to MySQL is to be SSL encrypted or if user identification is to take place with X509, you can specify the required information for establishing the connection here. The syntax is as follows:

    REQUIRE SSL | X509 [ISSUER 'iss' ] [SUBJECT 'subj'] [CIPHER 'ciph']

    REQUIRE SSL means that the connection must be SSL encrypted (thus a normal connection is not permitted). REQUIRE X509 means that the user must possess avalid certificate for identification that meets the X509 standard.

    ISSUER specifies the required issuer of the certificate. (Without ISSUER, the origin of the certificate is not considered.)

    SUBJECT specifies the required content of the certificate's subject field. (Without SUBJECT, the content is not considered.)

    CIPHER specifies the required SSL encryption algorithm. (SSL supports various algorithms. Without this specification, all algorithms are allowed, including older ones that may have security loopholes.)

  • maxlimits: Here you can specify how many connections per hour the user is allowed to establish, as well as the number of SELECT and INSERT/UPDATE/DELETE commands. The default setting for all three values is 0 (no limit):

    MAX_QUERIES_PER_HOUR n

    MAX_UPDATES_PER_HOUR n

    MAX_CONNECTIONS_PER_HOUR n

If the specified user does not yet exist and GRANT is executed without IDENTIFIED BY, then the new user has no password (which represents a security risk). On the other hand, if the user already exists, then GRANT without IDENTIFIED BY does not alter the password. (There is thus no danger that a password can be accidentally deleted by GRANT.)

It is impossible with GRANT to delete privileges that have already been granted (for example, by executing the command again with a smaller list of privileges). If you wish to take away privileges, you must use REVOKE.

GRANT may be used only by users with the Grant privilege. The user that executes GRANT may bestow only those privileges that he himself possesses. If the MySQL server is started with the option safe-user-create, then to create a new user, a user needs the Insert privilege for the table mysql.user in addition to the Grant privilege.

End example
Start example

HANDLER tablename OPEN [AS aliasname]
HANDLER tablename READ FIRST|NEXT [ WHERE condition LIMIT n, m ]
HANDLER tablename READ indexname FIRST|NEXT|PREV|LAST [ WHERE ...
LIMIT ...
HANDLER tablename CLOSE

Since MySQL 4.0, HANDLER enables direct access to MyISAM tables, and since MySQL 4.0.3, to InnoDB tables as well. This command can be used as a more efficient substitute for simple SELECT commands. This is particularly true if records are to be processed one at a time or in small groups.

The command is easy to use: First, access to a table is achieved with HANDLER OPEN. Then, HANDLER READ may be executed as often as you like, generally the first time with FIRST, and thereafter with NEXT, until no further results are forthcoming. The command returns results as with SELECT * (that is, all columns). HANDLER CLOSE terminates access.

HANDLER tablename READ reads the records in the order in which they were stored. On the other hand, the variant HANDLER tablename READ indexname uses the specified index. In my experiments, however, it has been impossible to use the primary index for this purpose, though it works with other indexes.

HANDLER was not conceived for use with typical MySQL applications, if for no other reason than that the code would be completely incompatible with every other database server. HANDLER is suitable for programming low-level tools (e.g., backup tools or drivers that simulate simple data access with a cursor). Note that HANDLER does not block tables (no locking), and therefore, the table can change while its data are being read.

End example
Start example

HELP function

Since MySQL 4.1, HELP returns a brief help text for the specified function.

INSERT [insertoptions] [INTO] tablename [ (columnlist) ]VALUES (valuelist1), ( ... )
...

INSERT [insertoptions] [INTO] tablename SET column1=value1, column2=value2
...

INSERT [insertoptions] [INTO] tablename [ (columnlist) ] SELECT ...

The INSERT command has the job of inserting new records into an existing table. There are three main syntax variants. In the first (and most frequently used) of these, new data records are specified in parentheses. Thus a typical INSERT command looks like this:

INSERT INTO tablename (columnA, columnB, columnC)
VALUES ('a', 1, 2), ('b', 7, 5)

The result is the insertion of two new records into the table. Columns that are allowed to be NULL, for which there is a default value, or which are automatically filled in by MySQL via AUTO_IN do not have to be specified. If the column names (i.e., in columnlist) are not given, then in VALUES all values must be given in the order of the columns.

With the second variant, only one record can be changed (not several simultaneously). Such a command looks like this:

INSERT INTO tablename SET columnA='a', columnB=1, columnC=2

For the third variant, the data come from a SELECT instruction.

  • insertoptions: The behavior of this command can be controlled with a number of global options:

    • IGNORE has the effect that the insertion of records with existing values is simply ignored for UNIQUE KEY columns. (Without this option, the result would be an error message.) ◦

    • LOW_PRIORITY | DELAYED have influence over when the insertion operation is carried out. In both cases, MySQL delays its storage operation until there are no pending read accesses to the table. The advantage of DELAYED is that MySQL returns OK at once, and the client does not need to wait for the end of the saving operation. However, DELAYED cannot be used if then an AUTO_INCREMENT value with LAST_INSERT_ID( ) is to be determined. DELAYED should also not be used if a LOCK was placed on the table. (The reason is this: For executing INSERT DELAYED,a new MySQL thread is started, and table locking uses threads in its operation.)

      The records to be inserted are stored in RAM until the insertion operation has actually been carried out. If MySQL should be terminated for some reason (crash, power outage), then the data are lost.

Please note that in using default values, there are special rules for TIMESTAMP and AUTO_INCREMENT values. (These rules hold for UPDATE commands as well.)

  • Columns with default values: If you want MySQL to use the default value for a column, then either do not specify this column in your INSERT command, or pass an empty character string (not NULL)as the value:

    INSERT INTO table (col1, col2_with_default_value) VALUES ('abc', '')

  • TIMESTAMP columns: If you want MySQL to insert the current time in the column, then either omit this column in your INSERT command, or pass the value NULL (not an empty character string). It is also allowed to pass a character string with a timestamp value if you wish to store a particular value.

  • AUTO_INCREMENT columns: Here as well, either you don't pass the column, or you pass the value NULL if MySQL is to determine the AUTO_INCREMENT value itself. You may pass any other value that is not otherwise in use.

End example
Start example

JOIN

JOIN is not actually an SQL command. This key word is mostly used as part of a SELECT command, to link data from several tables. JOIN will be described under SELECT.

End example
Start example

KILL threadid

This command terminates a specified thread (subprocess) of the MySQL server. It is allowed only to those users who possess the Super privilege. A list of running threads can be obtained via SHOW PROCESSLIST (where again, this command assumes the Process privilege). Threads can also be terminated via the external program mysqladmin.

End example
Start example

LOAD DATA [ loadoptions ] INFILE 'filename' [ duplicateoptions ]
INTO TABLE tablename
[ importoptions ]
[ IGNORE ignorenr LINES ]
[ (columnlist) ]

LOAD DATA reads a text file and inserts the data contained therein line by line into a table as data records. LOAD DATA is significantly faster then inserting data by multiple INSERT commands.

Normally, the file filename is read from the server's file system, on which MySQL is running. (For this, the FILE privilege is required. For security reasons, the file must either be located in the directory of the database or be readable by all users of the computer.)

  • loadoptions: LOCAL has the effect that the file filename on the local client computer is read (that is, the computer on which the command LOAD DATA is executed, not on the server computer). For this, no FILE privilege is necessary. (The FILE privilege relates only to the file system of the MySQL server computer.) Note that LOAD DATA LOCAL can be deactivated, depending on how the MySQL server was compiled and configured (option local-infile).

    LOW PRIORITY has the effect that the data are inserted into the table only if no other user is reading the table.

  • filename: If a file name is given without the path, then MySQL searches for this file in the directory of the current database (e.g., 'bulk.txt').

    If the file name is given with a relative path, then the path is interpreted by MySQL relative to the data directory (e.g., 'mydir/bulk.txt').

  • File names with absolute path are taken without alteration (for example, '/tmp/mydir/bulk.txt').

  • duplicateoptions: IGNORE | REPLACE determine the behavior of MySQL when a new data record has the same UNIQUE or PRIMARY KEY value as an existing record. With IGNORE, the existing record is preserved, and the new records are ignored. With REPLACE, existing records are replaced by the new ones. If neither of these options is used, then the result is an error message.

  • importoptions: Here is specified how the data should be formatted in the file to be imported. The entire importoptions block looks like this:

    [ FIELDS

    [ TERMINATED BY 'fieldtermstring' ]

    [ ENCLOSED BY 'enclosechar' ]

    [ ESCAPED BY 'escchar' ]]

    [ LINES TERMINATED BY 'linetermstring' ]

    • fieldtermstring specifies the character string that separates the individual columns within the row (e.g., a tab character).

    • enclosechar specifies the character that should stand before and after individual entries in the text file (usually the single or double quote character for character strings). If an entry begins with this character, then that character is removed from the beginning and end. Entries that do not begin with the enclosechar character will still be accepted. The use of the character in the text file is thus to some extent optional.

    • escchar specifies which character is to be used to mark special characters (usually the backslash). This is necessary if special characters appear in character strings in the text file that are also used to separate columns or rows. Furthermore, MySQL expects the zero-byte in the form \0, where the backslash is to be replaced as necessary by escchar if a character has been specified for escchar).

    • linetermstring specifies the character string with which rows are to be terminated. With DOS/Windows text files this must be the character string ' \r\n'.

In these four character strings, the following special characters can be specified:

\0 | code 0 | \t | tab
\b | backspace | \' | singe quote
\n | newline | \" | double quote
\r | carriage return | \\ | backslash
\s | blank character |  |

Furthermore, the character strings can be given in hexadecimal form (e.g., 0x22 instead of ' \'').

If no character strings are given, then the following is the default setting:

FIELDS TERMINATED BY ' \t' ENCLOSED BY '' ESCAPED BY ' \\'

LINES TERMINATED BY ' \n'

  • ignorenr: This value specifies how many lines should be ignored at the beginning of the text file. This is particularly useful if the first lines contain table headings.

  • columnlist: If the order of the columns in the text file does not exactly correspond to that in the table, then here one may specify which file columns correspond with which table columns. The list of columns must be set in parentheses: for example, (firstname, lastname, birthdate).

If TIMESTAMP columns are not considered during importation or if NULL is inserted, then MySQL inserts the actual time. MySQL exhibits analogous behavior with AUTO_INCREMENT columns.

LOAD DATA displays as result, among other things, an integer representing the number of warnings. Unfortunately, in MySQL 4.0 there is no way of determining the cause of the warnings (or at least the records or rows that caused the warnings). A typical cause is too many or too few columns in the file to be imported. Excess columns are simply ignored; for missing columns, either 0 or an empty character string is inserted. (An incorrect number of columns can also signify that MySQL was unable to locate the exact column boundaries.)

It looks as though starting with MySQL 4.1 you will be able to display all warnings and errors caused by LOAD DATA with the commands SHOW WARNINGS and SHOW ERRORS

Instead of LOAD DATA, you can also use the program mysqlimport. This program creates a link to MySQL and then uses LOAD DATA. The inverse of LOAD DATA is the command SELECT ... INTO OUTFILE. With it you can export a table into an text file. Further information and concrete examples can be found in Chapter 10.

End example
Start example

LOAD DATA FROM MASTER

This command (available since MySQL 4.0) copies all MyISAM tables from master to slave of a replication system. The tables of the mysql database are not copied. After copying, replication is begun on the slave (that is, the variables MASTER_LOG_FILE and MASTER_LOG_POS are set, which normally must be set with CHANGE MASTER TO).

This command can be used in many cases, in particular when no InnoDB tables are being used, for a convenient setting up of a replication system. It assumes that the replication user possesses the privileges Select, Reload, and Super.

End example
Start example

LOAD TABLE dbname.tablename FROM MASTER

This command copies a table in a replication system from master to slave, if the table does not yet exist there. The purpose of the command is actually to simplify debugging for MySQL developers. However, the command can possibly also be used for repairing a replication system after errors have been detected. The execution of the command requires that the replication user possess the privileges Select, Reload, and Super. LOAD TABLE works only for MyISAM tables.

End example
Start example

LOCK TABLE tablename1 [ AS aliasname ] locktype, table1 [ AS alias2 ] locktype, ...

LOCK TABLE prevents other MySQL users from executing write or read operations on the specified tables. If a table is already blocked by another user, then the command waits (unfortunately, without a timeout value, thus theoretically forever) until that block is released.

Table LOCK s ensure that during the execution of several commands no data are changed by other users. Typically, LOCK s are necessary when first a SELECT query is executed and then tables are changed with UPDATE, where the results of the previous query are used. (For a single UPDATE command, on the other hand, no LOCK is necessary. Individual UPDATE commands are always completely executed by the MySQL server without giving other users the opportunity to change data.)

LOCK TABLE should not be used on InnoDB tables, for which you can achieve much more efficient locking using transactions and with the commands SELECT ... IN SHARE MODE and SELECT ... FOR UPDATE.

  • locktype: Here one of four possible locking types must be specified.

    • READ: All MySQL users may read the table, but no one may change anything (including the user who executed the LOCK command). A READ LOCK is allocated only when the table is not blocked by other WRITE LOCK s. (Existing READ LOCK s, on the other hand, are no hindrance for new READ LOCK s. It is thus possible for several users to have simultaneous READ LOCK son the same table.)

    • READ LOCAL: Like READ, except that INSERTs are allowed if they do not change existing data records.

    • WRITE: The current user may read and change the table. All other users are completely blocked. They may neither change data in the blocked table nor read it. A WRITE LOCK is allocated only if the table is not blocked by other LOCK s (READ or WRITE). Until the WRITE LOCK is lifted, other users can obtain neither a READ LOCK nor a WRITE LOCK. ◦

    • LOW PRIORITY WRITE: Like WRITE, except that during the waiting time (that is, until all other READ and WRITE LOCK s have been ended) other users may obtain on demand a new READ LOCK. However, this means as well that the LOCK will be allocated only when there is no other user who wishes a READ LOCK.

Table LOCK s can increase the speed with which several database commands can be executed one after the other (of course, at the cost that other users are blocked during this time).

MySQL manages table LOCK s by means of a thread, where each connection is associated with its own thread. Only one LOCK command is considered per thread. (But several tables may be included.) As soon as UNLOCK TABLES or LOCK is executed for any other table, then all previous locks become invalid.

For reasons of efficiency, it should definitely be attempted to keep LOCK s as brief as possible and to end them as quickly as possible by UNLOCK. LOCK s end automatically when the current process ends (that is, for example, when the connection between server and client is broken).

End example
Start example

OPTIMIZE TABLE tablename

OPTIMIZE TABLE removes unused storage space from a table file and ensures that associated data in a data record are stored together. Currently (version 4.0.n), this functions only for MyISAM tables.

OPTIMIZE TABLE should be regularly executed for tables whose contents are continually being changed (many UPDATE and DELETE commands). Not only is the file made smaller, but access is speeded up. (Alternatively, the program myisamchk can be executed.)

End example
Start example

PROCEDURE procname

MySQL can be extended with procedures. However, their code must be formulated in the C++ programming language. To use such functions in SELECT commands, the key word PROCEDURE must be used.

As an example of such a procedure, the MySQL program code contains the function ANALYSE. This procedure can be used to analyze the contents of a table in the hope of determining a better table definition. The function is called thus:

SELECT * FROM tablename PROCEDURE ANALYSE()

As with the creation of user-defined functions (UDFs; see CREATE FUNCTION), the programming of procedures requires a great deal of MySQL background knowledge. Further information can be found in the MySQL documentation:

http://www.mysql.com/doc/en/Extending_MySQL.html

MySQL 5.0 plans on offering, in addition to UDFs, the possibility of creating one's own SQL functions and commands in the form of stored procedures (SPs). SPs are easier to program, but not as efficient as UDFs.

End example
Start example

PURGE MASTER LOGS TO 'hostname-bin.n'

This command deletes all binary logging files that are older than the file specified. Execute this command only when you are sure that the logging files are no longer needed, that is, when all slave computers have synchronized their databases. This command can be executed only on the master computer of a replication system, and only if the Super privilege has been granted. See also RESET MASTER.

End example
Start example

RENAME TABLE oldtablename TO newtablename

RENAME TABLE gives a new name to an existing table. It is also possible to rename several tables, e.g., aTOb,cTOd,etc.

In older versions of MySQL, RENAME TABLE is not available as a freestanding command. There you must use ALTER TABLE ... RENAME AS.

There is no command for giving a new name to an entire database. If you are using MyISAM tables, then to do so, you can stop the MySQL server, rename the database directory (mysql/data/dbname), and then restart the server. Note that you may have to change access privileges in the mysql database. With InnoDB tables, you must make a backup (mysqldump) and then import the tables into a new database.

End example
Start example

REPAIR TABLE tablename1, tablename2, ... [TYPE = QUICK ]

REPAIR TABLE attempts to repair a defective table file. With the option TYPE =QUICK only the index is re-created.

REPAIR TABLE can currently (version 4.0.n) be used only with MyISAM tables. Instead of this command, you may also use the external program myisamchk -r tblfile. (If REPAIR TABLE does not return OK as result, then you might try myisamchk -o. This program offers more repair possibilities than REPAIR TABLE.)

End example
Start example

REPLACE [INTO] see INSERT

REPLACE is a variant of INSERT. The only difference relates to new records whose key word is the same as that of an existing record. In this case, the existing record is deleted and the new one stored in the table. Since the behavior with duplicates is so clearly defined, REPLACE does not have the IGNORE option possessed by the INSERT command.

End example
Start example

RESET MASTER

This command deletes all binary logging files including the index file hostname-bin.index. With this command, replication can be restarted at a particular time. For this, RESET SLAVE must be executed on all slave systems. Before the command is executed it must be ensured that the databases on all slave systems are identical to those of the master system. This command assumes the reload privilege.

If you wish to delete only old (no longer needed) logging files, then use PURGE MASTER LOGS.

End example
Start example

RESET QUERY CACHE

This command deletes all entries from the query cache. It assumes the reload privilege.

End example

RESET SLAVE

This command reinitializes the slave system. The contents of master.info (and with it the current logging file and its position) are deleted. The command assumes the reload privilege.

This command makes sense only if after some problems the databases are to be set up on the slave based on previous snapshots so that the slave system then can synchronize itself by replication, or when RESET MASTER was executed on the master system (so that all logging files are deleted there). In this case, first SLAVE STOP and then SLAVE START should be executed on the slave system.

Start example

RESTORE TABLE tblname FROM '/backup/directory'

RESTORE TABLE copies the files of the specified table from a backup directory into the data directory of the current database. RESTORE TABLE is the inverse of BACKUP TABLE.

End example
Start example

REVOKE privileges ON objects FROM users

REVOKE is the inverse of GRANT. With this command you can remove individual privileges previously granted. The syntax for the parameters privileges, objects, and users can be read about under the GRANT command. The only difference relates to the Grant privilege: To revoke this privilege from a user, REVOKE can be used in the following form: REVOKE GRANT OPTION ON ... FROM ... .

Although GRANT inserts new users into the mysql.user table, REVOKE is incapable of deleting this user. You can remove all privileges from this user with REVOKE, but you cannot prevent this user from establishing a connection to MySQL. (If you wish to take that capability away as well, you must explicitly remove the entries from the user database with the DELETE command.)

Please note that in the MySQL access system you cannot forbid what is allowed at a higher level. If you allow x access to database d, then you cannot exclude table d.t with REVOKE. If you wish to allow x access to all tables of the database d with the exception of table t,then you must forbid access to the entire database and then allow access to individual tables of the database (with exception of d). REVOKE is not smart enough to carry out such operations on its own.

End example
Start example

ROLLBACK

ROLLBACK undoes the most recent transaction. (Instead of ROLLBACK, you can confirm the pending changes with COMMIT and thereby finalize their execution.)

BEGIN/COMMIT/ROLLBACK work only if you are working with transaction-capable tables. Further information and examples on transactions can be found in Chapter 8.

End example
Start example

SELECT [selectoptions] column1 [AS alias1], column2 [AS alias2] ...

SELECT [selectoptions] column1 [AS alias1], column2 [AS alias2] . . .

[ FROM tablelist ]

[ WHERE condition ]

[ GROUP BY groupfield [ASC | DESC] ]

[ HAVING condition ]

[ ORDER BY ordercolumn1 [DESC], ordercolumn2 [DESC] . . . ]

[ LIMIT [offset,] rows ]

[ PROCEDURE procname]

[LOCK IN SHARE MODE | FOR UPDATE ]

SELECT serves to formulate database queries. It returns the query result in tabular form. SELECT is usually implemented in the following form:

SELECT column1, column2, column3 FROM table ORDER BY column1

However, there are countless syntactic variants, thanks to which SELECT can be used also, for example, for processing simple expressions.

SELECT HOUR(NOW( ))

Note that the various parts of the SELECT command must be given in the order presented here.

  • selectoptions: The behavior of this command can be controlled by a number of options:

    • DISTINCT | ALL specify how MySQL should behave when a query returns several identical records. DISTINCT means that identical result records should be displayed only once. ALL means that all records should be displayed (the default setting).

    • Since MySQL 4.1, the sort order can also be specified for DISTINCT (which is also the basis for determining equivalence of character strings): The syntax is DISTINCT column COLLATE collname. ◦

    • HIGH_PRIORITY has the effect that a query with higher priority than change or insert commands will be executed. HIGH_PRIORITY should be used only for queries that need to be executed very quickly. ◦

    • SQL_BUFFER_RESULT has the effect that the result of a query is stored in a temporary table. This option should be used when the evaluation of the query is expected to range over a long period of time and locking problems are to be avoided during this period. ◦

    • SQL_CACHE and SQL_NO_CACHE specify whether the results of the SELECT command should be stored in the query cache or whether such storage should be prevented. By default, SQL_CACHE usually holds, unless the query cache is executed in demand mode (QUERY_CACHE_TYPE=2). ◦

    • SQL_CALC_FOUND_ROWS has the effect that MySQL determines the total number of found records even if you limit the result with LIMIT.

  • The number can then be determined with a second query SELECT FOUND_ROWS( ).

    • SQL_SMALL_RESULT | SQL_BIG_RESULT specify whether a large or small record list is expected as result, and they help MySQL in optimization. Both options are useful only with GROUP BY and DISTINCT queries. ◦

    • STRAIGHT_JOIN has the effect that data collected from queries extending over more than one table should be joined in the order of the FROM expression. (Without STRAIGHT_JOIN, MySQL attempts to find the optimal order on its own. STRAIGHT_JOIN bypasses this optimization algorithm.)

  • column: Here column names are normally given. If the query encompasses several tables, then the format is table.column. If a query is to encompass all the columns of the tables specified by FROM, then you can save yourself some typing and simply specify * . (Note, however, that this is inefficient in execution if you do not need all the columns.)

    However, instead of column names, you may also use general expressions or functions, e.g., for formatting a column (DATE_FORMAT( ... )) or for calculating an expression (COUNT( ... )).

    With AS,a column can be given a new name. This is practical in using functions such as HOUR(column) AS hr.

    Such an alias name can then be used in most of the rest of the SELECT command (e.g., ORDER BY hr). The alias name cannot, however, be placed in a WHERE clause.

    Since MySQL 4.1, the desired sort order can be specified with COLLATE (e.g., column COLLATE german1 AS alias).

  • tablelist: In the simplest case, there is simply a list (separated by commas) of all tables that are to be considered in the query. If no relational conditions (further below with WHERE) are formulated, then MySQL returns a list of all possible combinations of data records of all affected tables.

    There is the possibility of specifying here a condition for linking the tables, for example in the following forms:

    table1 LEFT [OUTER] JOIN table2 ON table1.xyID = table2.xyID

    table1 LEFT [OUTER] JOIN table2 USING (xyID)

    table1 NATURAL [LEFT [OUTER]] JOIN

    An extensive list of the many syntactic synonyms can be found in Chapter 6, where the topic of links among several tables is covered in great detail.

  • condition: Here is where conditions that the query results must fulfill can be formulated. Conditions can contain comparisons (column1>10 or column1=column2) or pattern expressions (column LIKE '%xy'), for example. Several conditions can be joined with AND, OR, and NOT.

    MySQL allows selection conditions with IN:

    WHERE id IN(1, 2, 3) corresponds to WHERE id=1 OR id=2 OR id=3. WHERE id NOT IN (1,2) corresponds to WHERE NOT (id=1 OR id=2).

    Sub SELECT s: Beginning with version 4.1, MySQL supports subSELECTs. This means that the results of one query can become part of the conditions of another. There are several syntactic variants:

    SELECT ... WHERE col = (SELECT ... ) SELECT ... WHERE col [NOT] IN (SELECT ... ) SELECT ... WHERE [NOT] EXISTS (SELECT ... )

    With the first variant, the second SELECT command must return a single value (e.g., SELECT MAX(col) FROM ... ); with the other variants, a list of values is possible. Examples of the use of subSELECTs can be found in Chapter 7 and Appendix B.

    Full-text search: Conditions can also be formulated with MATCH(col1, col2) AGAINST('word1 word2 word3'). Thereby a full-text search is carried out in the columns col1 and col2 for the words word1, word2, and word3. (This assumes that a full-text index for the columns col1 and col2 has been created.)

    AGAINST also supports Boolean search expressions, for example, in the form AGAINST('+word1 +word2 -word3' IN BOOLEAN MODE). Here the plus sign represents a logical AND operation, while the minus sign means that the specified word may not appear in the record. The full syntax for search expressions can be found in Chapter 7.

    WHERE versus HAVING: Conditions can be formulated with WHERE or HAVING. WHERE conditions are applied directly to the columns of the tables named in FROM.

    HAVING conditions, on the other hand, are applied only after the WHERE conditions to the intermediate result of the query. The advantage of HAVING is that conditions can also be specified for function results (for example, SUM(column1) in a GROUP BY query). Alias names can be used in HAVING conditions (AS xxx), which is not possible in WHERE.

    Conditions that can be equally well formulated with WHERE or HAVING should be expressed with WHERE, because in that case, better optimization is possible.

    groupfield: With GROUP BY, you can specify a group column. If the query returns several records with the same values for the group column, then these records are collected into a single new record. Along with GROUP BY, in the column part of the query so-called aggregate functions are usually placed, with which calculations can be made over grouped fields (e.g., COUNT, SUM, MIN, MAX).

    By default, grouped results are sorted as though ORDER BY had been specified for the columns. Optionally, the sort order can be determined with ASC or DESC.

    Since MySQL 4.1, the desired sort order can be given with COLLATE (e.g., GROUP BY column COLLATE german1).

  • ordercolumn: With ORDER BY several columns or expressions can be specified according to which the query result should be sorted. Sorting normally proceeds in increasing order (A, B, C, ... or 1, 2, 3, ... ). With the option DESC (for descending)you have decreasing order.

  • [offset,] row: With LIMIT the query results can be reduced to an arbitrary selection. This is to be recommended especially when the results are to be displayed pagewise or when the number of result records is to be limited. The position at which the results are to begin is given by offset (0 for the first data record), while row determines the maximum number of result records.

  • procname: This enables the call of a user-defined procedure (see the description of PROCEDURE).

If you use transactions, then the addition of LOCK IN SHARE MODE has the effect that all records found by the SELECT command will be blocked by a shared lock until the end of the transaction. This has two consequences: First, your SELECT query will not be executed until there is no running transaction that could change the result. Second, the affected records cannot now be changed by other connections (though they can be read with SELECT) until your transaction has ended.

FOR UPDATE is even more restrictive, blocking the found records with an exclusive lock. In contrast to a shared lock, other connections that execute SELECT ... LOCK IN SHARE MODE must now wait until the end of your transaction (of course, only if the same records are affected by SELECT queries).

MySQL does not currently support the formulation SELECT ... INTO table, known in many other SQL dialects. In most cases, you can use INSERT INTO ... SELECT or CREATE TABLE tablename ... SELECT ... .

End example
Start example

ROLLBACK TO SAVEPOINT name

ROLLBACK ends the current transaction, aborting all SQL commands that were executed after the specified SAVEPOINT. Commands up through SAVEPOINT are accepted. This command has been available since version 4.0.14.

End example
Start example

SAVEPOINT name

Since MySQL 4.0.14, this command places a marker in a running transaction. With ROLLBACK TO SAVEPOINT, the transaction can be recalled up to this point. SAVEPOINTs are valid only within a transaction, and they are deleted at the end of the transaction.

End example
Start example

SELECT [selectoptions] columnlist
INTO OUTFILE 'filename' exportoptions
[FROM ... WHERE ... GROUP BY ... HAVING ... ORDER BY ... LIMIT ... ]

With this variant of the SELECT command, the records are written into a text file. Here we describe only those options that are specific to this variant. All other points of syntax can be found under SELECT.

  • filename: The file is generated in the file system of the MySQL server. For security reasons, the file should not already exist. Moreover, you must have the FILE privilege to be able to execute this SELECT variant.

  • exportoptions: Here it is specified how the text file is formatted. The entire option block looks like this:

    [ FIELDS

    [ TERMINATED BY 'fieldtermstring' ]

    [ [OPTIONALLY] ENCLOSED BY 'enclosechar' ]

    [ ESCAPED BY 'escchar' ]]

    [ LINES TERMINATED BY 'linetermstring' ]

    • fieldtermstring specifies the character string that separates columns within a line (e.g., a tab character).

    • enclosechar specifies a character that is placed before and after every entry, e.g., '123' with ENCLOSED BY ' \''. With OPTIONALLY, the character is used only on CHAR, VARCHAR, TEXT, BLOB, TIME, DATE, SET, and ENUM columns (and not for every number format, such as TIMESTAMP).

    • escchar specifies the character to be used to mark special characters (usually the backslash). This is especially necessary when in character strings of a text file special characters appear that are also used for separating data elements.

    If escchar is specified, then the escape character is always used for itself (\\) as well as for ASCII code 0 (\0). If enclosechar is empty, then the escape character is also used as identifier of the first character of fieldtermstring and linetermstring (e.g., \t and \n). On the other hand, if enclosechar is not empty, then escchar is used only for enclosechar (e.g., \" ), and not for fieldtermstring and linetermstring. (This is no longer necessary, since the end of the character string is uniquely identifiable due to enclosechar.)

    • linetermstring specifies the character string with which lines are to be terminated. With DOS/Windows text files this must be the character string ' \r\n'.

In the four character strings, special characters can be specified, for example, \b for backspace. The list of permissible special characters can be found at the command LOAD DATA.Moreover, character strings can be given in hexadecimal notation (such as 0x22 instead of ' \'').

As with LOAD DATA the following is the default setting:

FIELDS TERMINATED BY ' \t' ENCLOSED BY '' ESCAPED BY ' \\'

LINES TERMINATED BY ' \n'

If you wish to input files generated with SELECT ... INTO OUTFILE again into a table, then use LOAD DATA. This command is the inverse of SELECT ... INTO OUTFILE. Further information and concrete application examples for both commands can be found in Chapter 10.

End example
Start example

SELECT [selectoptions] column

INTO DUMPFILE 'filename'

[ FROM . . . WHERE . . . GROUP BY . . . HAVING . . . ORDER BY . . . LIMIT . . . ]

SELECT ... INTO DUMPFILE has, in principle, the same function as SELECT ... INTO OUTFILE (see above). The difference is that here data are stored without any characters to indicate column or row division.

SELECT ... INTO DUMPFILE is designed for saving a single BLOB object into a file. The SELECT query should therefore return precisely one column and one row as result. Should that not be the case, that is, if the query returns more than one data element, then usually (and for some strange reason not always) one receives an error message: ERROR 1172: Result consisted of more than one row.

End example
Start example

(SELECT selectoptions) UNION [ALL](SELECT selectoptions) unionoptions

Since MySQL 4.0 you can use UNION to unite the results of two or more SELECT queries. You thereby obtain a result table in which the results of the individual queries are simply strung together. The individual queries can affect different tables, though you must ensure that the number of columns and their data types are the same.

The optional key word ALL has the effect that duplicates (that is, results that arise in more than one SELECT query) appear in the end result with their corresponding multiplicity. Without ALL, duplicates are eliminated (as with DISTINCT in SELECT).

With SELECT commands, all options described earlier for selecting columns, setting sort order, etc., are permitted. With unionoptions you can also specify how the final result is to be sorted (ORDER BY ) and reduced (LIMIT).

End example
Start example

SET @variable = expression

MySQL permits the management of one's own user variables. These variables are indicated by the @ symbol before the name. These variables are managed separately for each client connection, so that no naming conflicts can arise among clients. The content of such variables is lost at the end of the connection.

Instead of SET, one may also use SELECT for the assignment of user variables. The syntax is SELECT @variable:=expression (note that := must be used instead of =).

End example
Start example

SET [options] [@@]systemvariable = expression

If the variable name has either no @ prefixed or two of them (@@), then SET is setting system variables.

  • options: MySQL distinguishes two levels of validity among system variables: GLOBAL (valid for the entire MySQL server) and SESSION (valid only for the current connection). The default setting is SESSION.

Variables at the global level can be changed only by users with the Super privilege. Global changes are valid only for new connections, not those already in existence.

End example
Start example

SET [OPTION] option=value

SET can also used to modify certain MySQL options as well as the password. Although the syntax looks the same as that for variable assignment, here we are not dealing with system variables that can be evaluated with SELECT @@name or SHOW VARIABLES.

For example, with

SET SQL_LOW_PRIORITY_UPDATES=0/1

it is possible to determine the order in which MySQL executes queries and change commands. The default behavior (1) gives priority to change commands. (This has the effect that a lengthy SELECT command will not block change commands, which are usually executed quickly.) With the setting 0, on the other hand, changes are executed only when no SELECT command is waiting to be executed.

SET PASSWORD also offers a convenient way of changing one's password (and saves the relatively more cumbersome manipulation of the access table of the mysql database:

SET PASSWORD = PASSWORD('some password')

If you have sufficient privileges, you can use SET to set the password of another user:

SET PASSWORD FOR username@hostname = PASSWORD('newPassword')

SET AUTOCOMMIT = 0 or 1 switches the autocommit mode for transactions off or on. Autocommit mode holds only for transaction-capable tables (see Chapter 8).

If you use replication, you can temporarily interrupt binary logging on the master system with SET SQL_LOG_BIN =0 in order to make manual changes that should not be replicated. SET SQL_LOG_BIN=1 resumes logging.

SET SQL_QUERY_CACHE = 0|1|2|ON|OFF|DEMAND sets the mode of the query cache (see Chapter 10).

SET TRANSACTION ISOLATION LEVEL sets the isolation level for transactions. The setting holds for transaction-capable tables. Here is the syntax:

SET [SESSION|GLOBAL] TRANSACTION ISOLATION LEVEL
  READ UNCOMMITTED | READ COMMITTED |
  REPEATABLE READ | SERIALIZABLE

SET SESSION changes the transaction degree for the current connection, and SET GLOBAL for all future connections (but not the current one). If neither SESSION nor GLOBAL is specified, then the setting is valid only for the coming transaction. (Note that SESSION and GLOBAL in SET TRANSACTION have a somewhat different effect from that of SET [@@]systemvariable.)

The four isolation degrees are described in Chapter 8. With InnoDB tables, the default is REPEATABLE READ. The isolation degree can also be read from the variable @@[global.]tx_isolation.

Additional SET options: The following list presents all the options that can be changed with SET:

SET BIG_TABLES = 0 | 1
SET CHARACTER SET character_set_name | DEFAULT
SET INSERT_ID = #
SET LAST_INSERT_ID = #
SET LOW_PRIORITY_UPDATES = 0 | 1
SET MAX_JOIN_SIZE = value | DEFAULT
SET QUERY_CACHE_TYPE = 0 | 1 | 2
SET QUERY_CACHE_TYPE = OFF | ON | DEMAND
SET SQL_AUTO_IS_NULL = 0 | 1
SET SQL_BIG_SELECTS = 0 | 1
SET SQL_BUFFER_RESULT = 0 | 1
SET SQL_LOG_OFF = 0 | 1
SET SQL_LOG_UPDATE = 0 | 1
SET SQL_QUOTE_SHOW_CREATE = 0 | 1
SET SQL_SAFE_UPDATES = 0 | 1
SET SQL_SELECT_LIMIT = value | DEFAULT
SET TIMESTAMP = timestamp_value | DEFAULT

A description of these (mostly seldom used) setting options can be found in the MySQL documentation in the description of the SET command:

http://www.mysql.com/doc/en/SET_OPTION.html
End example
Start example

SHOW BINLOG EVENTS [IN logname] [FROM pos] [LIMIT offset, rows]

If this command is executed without options, then it returns the complete contents of the currently active binary logging file. The options allow for the specification of other logging files or for limiting the output. Note that this command can also be used to read the logging file of an external MySQL server.

End example
Start example

SHOW CHARACTER SET [LIKE pattern ]

Since MySQL 4.1, SHOW CHARACTER SET returns a list of all available character sets and their default sort orders.

End example
Start example

SHOW COLLATION[ LIKE pattern ]

Since MySQL 4.1, SHOW COLLATION returns a list of all available sort orders.

End example
Start example

SHOW COLUMN TYPES

Since MySQL 4.1, SHOW COLUMN TYPES returns a list of all data types available for column definition. (In the tested alpha version this did not yet work.)

One may store results of queries in user variables. However, the storage of several values is not possible:

SELECT @a:=SUM(colA) FROM table1

User variables can be placed, for example, into WHERE conditionals:

SELECT * FROM table2 WHERE colA>@a
End example
Start example

SHOW [FULL] COLUMNS FROM tablename [ FROM databasename][ LIKE pattern ]

SHOW COLUMNS returns a table with information on all columns of a table (field name, field type, index, default value, etc.). With LIKE the list of columns can be filtered with a search pattern with the wild cards _ and %. The optional key word FULL has the effect that the access privileges of the current user are also displayed on the columns. The same information can be obtained with SHOW FIELDS FROM tablename, EXPLAIN tablename, or DESCRIBE tablename, as well as with the external program mysqlshow.

End example
Start example

SHOW CREATE DATABASE tablename

Since MySQL 4.1, SHOW CREATE DATABASE displays the SQL command with which the specified database can be re-created.

End example
Start example

SHOW CREATE TABLE tablename

SHOW CREATE TABLE displays the SQL command with which the specified table can be re-created.

End example
Start example

SHOW DATABASES [ LIKE pattern ]

SHOW DATABASES returns a list of all databases that the user can access. The list can be filtered with a search pattern with the wild cards _ and %. The same information can also be obtained with the external program mysqlshow.

For users possessing the Show Databases privilege, SHOW DATABASES returns a list of all databases, including those to which the user does not have access.

End example
Start example

SHOW [COUNT(*)] ERRORS

Since MySQL 4.1, SHOW ERRORS returns a list of errors that were triggered by the execution of the most recent command.

End example
Start example

SHOW FIELDS see SHOW COLUMNS

End example
Start example

SHOW GRANTS FOR user@host

SHOW GRANTS displays a list of all access privileges for a particular user. It is necessary that user and host be specified exactly as these character strings are stored in the various mysql access tables. Wild cards are not permitted.

End example
Start example

SHOW INDEX FROM table

SHOW INDEX returns a table with information about all indexes of the given table.

End example
Start example

SHOW INNODB STATUS

SHOW INNODB STATUS returns information about various internal workings of the InnoDB table driver. The data can be used for speed optimization (see the section Performance Tuning Tips in the InnoDB documentation).

End example
Start example

SHOW KEYS see SHOW INDEX

End example
Start example

SHOW LOGS

This command shows which BDB logging files are currently being used. (If you are not using BDB tables, the command returns no result.)

End example
Start example

SHOW MASTER LOGS

This command returns a list of all binary logging files. It can be executed only on the master computer of a replication system.

End example
Start example

SHOW MASTER STATUS

This command shows which logging file is the current one, as well as the current position in this file and which databases are excepted from logging (configuration settings binlog-do-db and binlog-ignore-db). This command can be used only on the master computer of a replication system.

End example
Start example

SHOW PRIVILEGES

Since MySQL 4.1, this command returns a list of all available privileges with a brief description.

End example
Start example

SHOW [ FULL ] PROCESSLIST

This command returns a list of all running threads (subprocesses) of the MySQL server. If the PROCESS privilege has been granted, then all threads are shown. Otherwise, only the user's threads are displayed.

The option FULL has the effect that for each thread, the complete text of the most recently executed command is displayed. Without this option, only the first 100 characters are shown.

The process list can also be determined with the external command mysqladmin.

End example
Start example

SHOW SLAVE HOSTS

This command returns a list of all slaves that replicate the master's databases. The command can be used only by the master computer of a replication system. It functions only for slaves for which the host name is specified in the configuration file explicitly in the form report-host=hostname.

End example
Start example

SHOW SLAVE STATUS

This command provides information on the state of replication, including the display of all information about the file master.info. This command can be executed only on a slave computer in a replication system.

End example
Start example

SHOW STATUS

This command returns a list of various MySQL variables that provide information on the current state of MySQL (for example, Connections, Open_files, Uptime). This same information can also be determined with the external program mysqladmin. A description of all variables can be found in the MySQL documentation under the command SHOW STATUS:

http://www.mysql.com/doc/en/SHOW_STATUS.html
End example
Start example

SHOW TABLE STATUS [FROM database] [LIKE pattern]

SHOW TABLE STATUS returns information about all tables of the currently active or specified database: table type, number of records, average record length, Create_time, Update_time, etc. The same information can also be determined with the external program mysqlshow.With pattern the list of tables can be limited; the SQL wild cards % and _ are permitted in pattern.

End example
Start example

SHOW TABLE TYPES

Since MySQL 4.1, SHOW TABLE TYPES returns a list of all available table types (MyISAM, HEAP, InnoDB, etc.).

End example
Start example

SHOW TABLES [FROM database] [LIKE pattern]

SHOW TABLES returns a list of all tables of the current (or specified) database. Optionally, the list of all tables can be reduced to those matching the search pattern pattern (where the SQL wild cards % and _ are allowed). More information on the construction of individual tables can be obtained with DESCRIBE TABLE and SHOW COLUMNS. The list of tables can also be retrieved with the external program mysqlshow.

End example
Start example

SHOW [options] VARIABLES [LIKE pattern]

This command returns a seemingly endless list of all system variables defined by MySQL together with their values (e.g., ansi_mode, sort_buffer, tmpdir, wait_timeout, to name but a very few). To limit the list, a pattern can be given (e.g., LIKE 'char%').

Many of these variables can be set at launch of MySQL or afterwards with SET. The list of variables can also be recovered with the external command mysqladmin.

  • options: Here you can specify GLOBAL or SESSION. GLOBAL has the effect that the default values valid at the global level are displayed. SESSION, on the other hand, results in the values being displayed that are valid for the current connection. The default is SESSION.

An extensive description of the variables can be found in the MySQL documentation:

http://www.mysql.com/doc/en/SHOW_VARIABLES.html
End example
Start example

SHOW [COUNT(*)] WARNINGS

Since MySQL 4.1, this command returns a list of all warnings that arose from the execution of the most recent command.

End example
Start example

SLAVE START/STOP [IO_THREAD | SQL_THREAD]

These commands start and stop replication (we leave it to the reader to determine which is which). They can be executed only on the slave computer of a replication system.

By default, two threads are started for replication: the IO thread (copies the binary logging data from the master to the slave) and the SQL thread (executes the logging file's SQL command). With the optional specification of IO_THREAD or SQL_THREAD, these two threads can be started or stopped independently (which makes sense only for debugging).

End example
Start example

START TRANSACTION

If you are working with transaction-capable tables, START TRANSACTION initiates a new transaction. The command is ANSI-99 conforming, but it has been available only since MySQL 4.0.11. In earlier versions, you must use the equivalent command BEGIN.

End example
Start example

TRUNCATE TABLE tablename

TRUNCATE has the same functionality as DELETE without a WHERE condition; that is, the effect is that all records in the table are deleted. This is accomplished by deleting the entire table and then recreating it. (This is considerably faster than deleting each record individually.)

TRUNCATE cannot be part of a transaction. TRUNCATE functions like COMMIT; that is, all pending changes are first executed. TRUNCATE can also be undone with ROLLBACK.

End example
Start example

UNION see SELECT UNION

With UNION,you can assemble the results of several SELECT queries.

End example
Start example

UNLOCK TABLES

UNLOCK TABLES removes all of the user's LOCK s. This command holds for all databases (that is, it doesn't matter which database is the current one).

End example
Start example

UPDATE [updateoptions] tablename
SET col1=value1, col2=value2, ...
[ WHERE condition ] [ ORDER BY columns ] [ LIMIT maxrecords ]

UPDATE changes individual fields of the table records specified by WHERE. Those fields not specified by SET remain unchanged. In value one can refer to existing fields. For example, an UPDATE command may be of the following form:

UPDATE products SET price = price+5 WHERE productID=3

Warning: Without a WHERE condition, all data records in the table will be changed. (In the above example, the prices of all products would be increased by 5.)

  • updateoptions: Here the options LOW PRIORITY and IGNORE may be given. The effect is the same as with INSERT.

  • condition: This condition specifies which records are affected by the change. For the syntax of condition see SELECT.

  • columns: With ORDER BY, you can sort the record list before making changes. This makes sense only in combination with LIMIT, for example, to change the first or last ten records (ordered according to some criterion). This possibility has existed since MySQL 4.0.

  • maxrecords: With LIMIT, the maximum number of records that may be changed is specified.

End example
Start example

UPDATE [updateoptions] table1, table2, table3
SET table1.col1=table2.col2 ...
[ WHERE condition][ORDER BY columns ] [ LIMIT maxrecords ]

Since MySQL 4.0, UPDATE commands can include more than one table. All tables included in the query must be specified after UPDATE. The only tables that are changed are those whose fields were specified by SET. The link between the tables must be set with WHERE conditions.

End example
Start example

USE databasename

USE turns the specified database into the default database for the current connection to MySQL. Until the end of the connection (or until the next USE command), all table names are automatically assigned to the database databasename.

End example


Function Reference

The functions described here can be used in SELECT queries are well as in other SQL commands. We begin with a few examples. In our first example, we shall join two table columns with CONCAT to create a new character string. In the second example, the function PASSWORD will be used to store an encrypted password in a column. In the third example, the function DATE_FORMAT will be summoned to help us format a date:

SELECT CONCAT(firstname, ' ', lastname) FROM users
  Peter Smith
   ...
INSERT INTO logins (username, userpassword)
VALUES ('smith', PASSWORD('xxx'))
SELECT DATE_FORMAT(a_date, '%Y %M %e')
FROM exceptions.test_date
    2003 December 7
Pointer

This section aims to provide only a compact overview of the functions available. Extensive information on these functions can be found in the MySQL documentation. Some of these functions have been introduced at various places in this book by way of example. See the Index for page numbers.

Arithmetic Functions

ABS(x) | calculates the absolute value (nonnegative number)
ACOS(x), ASIN(x) | calculates the arcsin and arccos
ATAN(x), ATAN2(x, y) | calculates the arctangent
CEILING(x) | rounds up to the least integer greater than or equal to x
COS(x) | calculates the cosine; x is given in radians
COT(x) | calculates the cotangent
DEGREES(x) | converts radians to degrees (multiplication by 180/p)
EXP(x) | returns ex
FLOOR(x) | rounds down to the greatest integer less than or equal to x
LOG(x) | returns the natural logarithm (i.e., to base e)
LOG10(x) | returns the logarithm to base 10
MOD(x, y) | returns the mod function, equivalent to x % y
PI( ) | returns 3.1415927
POW(x, y) | returns xy
POWER(x, y) | corresponds to POW(x, y)
RADIANS(x) | converts degrees into radians (multiplication by p/180)
RAND( ) | returns a random number between 0.0 and 1.0
RAND(n) | returns a reproducible (thus not quite random) number
ROUND(x) | rounds to the nearest integer
ROUND(x, y) | rounds to y decimal places
SIGN(x) | returns - 1, 0, or 1 depending on the sign of x
SIN(x) | calculates the sine
SQRT(x) | calculates the square root
TAN(x) | calculates the tangent
TRUNCATE(x) | removes digits after the decimal point
TRUNCATE(x, y) | retains y digits after the decimal point (thus TRUNCATE(1.236439, 2) returns 1.23)

In general, all functions return NULL if provided with invalid parameters (e.g., SQRT(-1)).

Comparison Functions, Tests, Branching

Comparison Functions

COALESCE(x, y, z . . . ) | returns the first parameter that is not NULL
GREATEST(x, y, z, . . . ) | returns the greatest value or greatest character string
IF(expr, val1, val2) | returns val1 if expr is true; otherwise, val2
IFNULL(expr1, expr2) | returns expr2 if expr1 is NULL; otherwise, expr1
INTERVAL(x, n1, n2, . . . ) | returns 0 if x < n1; 1 if x < n2, etc.; all parameters must be integers, and n1 < n2 < · · · must hold
ISNULL(x) | returns 1 or 0, according to whether x IS NULL holds
LEAST(x, y, z, . . . ) | returns the smallest value or smallest character string
STRCMP(s1, s2) | returns 0 if s1 = s2 in sort order, - 1 if s1 < s2, 1 if s1 > s2.

Tests, Branching

IF(expr, result1, result2) | returns result1 if expr is true; otherwise, result2
CASE expr | returns result1 if expr=val1, returns result2 if expr=val2,
WHEN val1 THEN result1 | etc. If no condition is satisfied, then the result is resultn
WHEN val2 THEN result2 |
ELSE resultn |
END |
CASE | returns result1 if condition cond1 is true, etc.
WHEN cond1 THEN result1 |
WHEN cond2 THEN result2 |
ELSE resultn |
END |

Type Conversion (CAST)

CAST(x AS type) | transforms x into the specified type; CAST can deal with the following types: BINARY, CHAR, DATE, DATETIME, SIGNED [INTEGER], TIME, UNSIGNED [INTEGER]
CONVERT(x, type) | equivalent to CAST(x AS type)
CONVERT(s USING cs) | represents the string s in the character set cs (since MySQL 4.1)

String Processing

Most character string functions can also be used for processing binary data. Since MySQL 4.1 (with Unicode support), the position and length specification functions such as LEFT and MID apply to characters, not bytes. MID(column, 3, 1) thus returns the third character, regardless of the character set that is defined for column.

Processing Character Strings

CHAR_LENGTH(s) | returns the number of characters in s; CHAR_LENGTH works also for multibyte character sets (e.g., Unicode)
CONCAT(s1, s2, s3, . . . ) | concatenates the strings
CONCAT_WS(x, s1, s2, . . . ) | functions like CONCAT, except that x is inserted between each string; CONCAT_WS(', ', 'a', 'b', 'c') returns 'a, b, c'
ELT(n, s1, s2, . . . ) | returns the nth string; ELT(2, 'a', 'b', 'c') returns 'b'
EXPORT_SET(x, s1, s2) | creates a string from s1 and s2 based on the bit coding of x; x is interpreted as a 64-bit integer
FIELD(s, s1, s2, . . . ) | compares s with strings s1, s2 and returns the index of the first matching string; FIELD('b', 'a', 'b', 'c') returns 2
FIND_IN_SET(s1, s2) | searches for s1 in s2; s2 contains a comma-separated list of strings; FIND_IN_SET('b', 'a,b,c') returns 2
INSERT(s1, pos, 0, s2) | inserts s2 into position pos in s1;
INSERT('ABCDEF', 3, 0, 'abc') | returns 'ABabcDEF'
INSERT(s1, pos, len, s2) | inserts s2 at position pos in s1 and replaces len characters of s2 with the new characters; INSERT('ABCDEF', 3, 2, 'abc') returns 'ABabcEF'
INSTR(s, sub) | returns the position of sub in s;
INSTR('abcde', 'bc') | returns 2
LCASE(s) | changes uppercase characters to lowercase
LEFT(s, n) | returns the first n characters of s
LENGTH(s) | returns the number of bytes necessary to store the string s; if multibyte character sets are used (e.g., Unicode), then CHAR_LENGTH must be used to determine the number of characters
LOCATE(sub, s) | returns the position of sub in s; LOCATE('bc', 'abcde') returns 2
LOCATE(sub, s, n) | as above, but the search for sub begins only at the nth character of s
LOWER(s) | transforms uppercase characters to lowercase
LPAD(s, len, fill) | inserts the fill character fill into s, so that s ends up with length len; LPAD('ab', 5, '*') returns '***ab'
LTRIM(s) | removes spaces at the beginning of s
MAKE_SET(x, s1, s2 . . . ) | forms a new string in which all strings sn appear for which in x the bit n is set; MAKE_SET(1+2+8, 'a', b', 'c', 'd') returns 'a,b,d'
MID(s, pos, len) | reads len characters from position pos from the string s; MID('abcde', 3, 2) returns 'cd'
POSITION(sub IN s) | equivalent to LOCATE(sub, s)
QUOTE(s) | since MySQL 4.0, returns a string suitable for SQL com- mands; special characters such as ', ", \ are prefixed with a backspace
REPEAT(s, n) | joins s to itself n times; REPEAT('ab', 3) returns 'ababab'
REPLACE(s, fnd, rpl) | replaces in s all fnd strings by rpl; REPLACE('abcde', 'b', 'xy') returns 'axycde'
REVERSE(s) | reverses the string
RIGHT(s, n) | returns the last n characters of s
RPAD(s, len, fill) | inserts the fill character fill at the end of s, so that s has length len; RPAD('ab', 5, '*') returns 'ab***'
RTRIM(s) | removes spaces from the end of s
SPACE(n) | returns n space characters
SUBSTRING(s, pos) | returns the right part of s from position pos
SUBSTRING(s, pos, len) | as above, but only len characters (equivalent to MID(s, pos, len))
SUBSTRING_INDEX(s, f, n) | searches for the nth appearance of f in s and returns the left part of the string up to this position (exclusive); for negative n, the search begins at the end of the string, and the right part of the string is returned;
 | SUBSTRING_INDEX('abcabc', 'b', 2) returns 'abca'
 | SUBSTRING_INDEX('abcabc', 'b', -2) returns 'cabc'
TRIM(s) | removes spaces from the beginning and end of s
TRIM(f FROM s) | removes the character f from the beginning and end of s
UCASE(s) / UPPER(s) | transforms lowercase characters to uppercase

Converting Numbers and Character Strings

ASCII(s) | returns the byte code of the first character of s: thus ASCII('A') returns 65; see also ORD
BIN(x) | returns the binary code of x; BIN(12) returns '1010'
CHAR(x, y, z, . . . ) | returns the string formed from the code x, y, . . . ; CHAR(65, 66) returns 'AB'
CHARSET(s) | since MySQL 4.1, returns the name of the character set in which s is represented
CONV(x, from, to) | transforms x from number base from to base to; CONV(25, 10, 16) returns the hexadecimal '19'
CONVERT(s USING cs) | since MySQL 4.1, represents the string s in the character set cs
FORMAT(x, n) | formats x with commas for thousands separation and n decimal places; FORMAT(12345.678, 2) returns '12,345.68'
HEX(x) | returns the hexadecimal code for x; x can be a 64-bit integer or (since MySQL 4.0) a character string; in the second case, each character is transformed into an 8-bit hex code; HEX('abc') returns '414243'
INET_NTOA(n) | transforms n into an IP address with at least four groups; INET_NTOA(1852797041) returns '110.111.112.113' INET_ATON(ipadr) transforms an IP address into the corresponding 32- or 64-bit integer; INET_ATON('110.111.112.113') returns 1852797041
OCT(x) | returns the octal code of x
ORD(s) | like ASCII(s), returns the code of the first character, but functions also for multibyte character sets
SOUNDEX(s) | returns a string that should match similar-sounding English words; SOUNDEX('hat') and SOUNDEX('head') both return 'H300'; extensive information on the SOUNDEX algorithm can be found in the book SQL for Smarties by Joe Celko

Check Sums

CRC32(s) | since MySQL 4.1, computes a check (cyclic redundancy check value) for the string s
MD5(str) | computes the MD5 check sum for the string str
SHA(str), SHA1(str) | since MySQL 4.0, computes a 160-bit check sum using the SHA1 algorithm (defined in RFC 3174). SHA is considered more secure than MD5. The result is returned as a string containing a 40-digit hexadecimal code; SHA and SHA1 are synonyms.

Encryption of Character Strings and Password Management

AES_DECRYPT(crypt, key) | decrypts crypt since MySQL 4.0 with the AES algorithm (Rijndael) and uses key for decryption
AES_ENCRYPT(str, key) | encrypts str using key for encryption
DES_DECRYPT(crypt [, keyno | | decrypts crypt using the DES algorithm; available since
keystr]) | MySQL 4.0 and only when MySQL is compiled with SSL functions; without the optional second parameter, the first key form the DES key file is used for encryption; optionally, the number or name of the key can be specified
DES_ENCRYPT(str [, keyno | keystr]) | encrypts str using the DES algorithm
DECODE(crypt, pw) | decrypts crypt using the password pw
ENCODE(str, pw) | encrypts str using pw as password; the result is a binary object that can be decrypted with DECODE
ENCRYPT(pw) | encrypts the password with the UNIX crypt function; if this function is unavailable, returns ENCRYPT NULL
PASSWORD(pw) | encrypts the password with the algorithm that was used for storing passwords in the USER table; the result is a 16-character string; note that since MySQL 4.1, PASSWORD uses stronger encryption and returns a string of 45 characters
OLD_PASSWORD(pw) | encrypts the password as for PASSWORD under MySQL 3.23.n and 4.0.n; available since MySQL 4.1

Date and Time

There are various synonyms for the functions in the following table, CURRENT_TIMESTAMP, for example, as an alternative to NOW( ), which have not been listed. The functions for dealing with dates and times generally assume that the initial data are correct. Thus do not expect a sensible result if you provide the date '2003-02-31'.

With all functions that return a time or date (or both), the format of the result depends on the context. Normally, the result is a character string (e.g., '2003-12-31 23:59:59'). However, if the function is used in a numerical computation, such as NOW( )+0, then the result is an integer of the form 20031231235959.

Several of the functions listed in the table process Unix timestamps, and there is a MySQL data type for such timestamps. What, then, is a timestamp?

In Unix, time is counted for timestamps from 1 January 1970. Each second since then is counted. Since the (signed) integer used for this number was declared as a 32-bit integer, it will overflow in the year 2038. For the time period from 1970 to 2038, then, a timestamp is a practical affair: compact and simple to use. But then what?

First of all, there is still one bit in reserve if the timestamp is viewed as an unsigned integer. On this assumption we have until the beginning of the twenty-second century until overflow occurs. But on the other hand, it is expected that in the next few years, Unix/Linux will obtain a 64-bit timestamp. At that point, presumably, MySQL will be reconfigured. Because of the increased memory requirement (eight bytes instead of four), all tables will have to migrate to the new format. Presumably, there will be no problems with this, and the year 2000 problem will not repeat itself in 2038 (at least with respect to MySQL).

Using Dates and Times

CURDATE( ) | returns the current date in the form '2003-12-31'
CURTIME( ) | returns the current time either as a string or number, depending on context; e.g., '23:59:59' or 235959 (integer)
DATE_ADD( . . . ) | adds a time interval to a starting time; see below
DATE_FORMAT(d, form) | formats d according to formatting string f ; see below
DATE_SUB( . . . ) | subtracts a time interval from the start time; see below
DAYNAME(date) | returns 'Monday', 'Tuesday', etc.
DAYOFMONTH(date) | returns the day of the month (1–31)
DAYOFWEEK(date) | returns the day of the week (1 = Sunday through 7 = Saturday)
DAYOFYEAR(date) | returns the day in the year (1–366)
EXTRACT(i FROM date) | returns a number for the desired interval
EXTRACT(YEAR FROM '2003- 12-31') | returns 2003
FROM_DAYS(n) | returns the date n days after the year 0
FROM_DAYS(3660) | returns '0010-01-08'
FROM_UNIXTIME(t) | transforms the Unix timestamp number t into a date
FROM_UNIXTIME(0) | returns '1970-01-01 01:00:00'
FROM_UNIXTIME(t, f) | as above, but with formatting as in DATE_FORMAT
HOUR(time) | returns the hour (0–23)
MINUTE(time) | returns the minute (0–59)
MONTH(date) | returns the month (1–12)
MONTHNAME(date) | returns the name of the month ('January', etc.)
NOW( ) | returns the current time in the form '2003-12-31 23:59:59'
QUARTER(date) | returns the quarter (1–4)
SECOND(time) | returns the second (0–59)
SEC_TO_TIME(n) | returns the time n seconds after midnight
SEC_TO_TIME(3603) | returns '01:00:03'
TIME_FORMAT(time, f) | like DATE_FORMAT, but for times only
TIME_TO_SEC(time) | returns the seconds since midnight
TO_DAYS(date) | returns the number of days since the year 0
UNIX_TIMESTAMP( ) | returns the current time as a Unix timestamp number
UNIX_TIMESTAMP(d) | returns the timestamp number for the given date
WEEK(date) | returns week number (1 for the week beginning with the first Sunday in the year)
WEEK(date, day) | as above, but specifies with day the day on which the week should begin (0 = Sunday, 1 = Monday, etc.)
WEEKDAY(date) | returns the day of the week (0 = Monday, 1 = Tuesday, etc.)
YEAR(date) | returns the year

Calculating with Dates

DATE_ADD(date, INTERVAL n i) adds n times the interval i to the starting date date. Our first example shows how intelligently the function deals with ends of months (31.12 or 28.2):

DATE_ADD('2003-12-31', INTERVAL 2 month) returns '2004-02-28'
DATE_ADD('2003-12-31', INTERVAL '3:30' HOUR_MINUTE) returns '2003-12-31 03:30:00'

Intervals for DATE_ADD, DATE_SUB, EXTRACT

SECOND | n
MINUTE | n
HOUR | n
DAY | n
MONTH | n
YEAR | n
HOUR_MINUTE | 'hh:mm'
HOUR_SECOND | 'hh:mm:ss'
MINUTE_SECOND | 'mm:ss'
DAY_HOUR | 'dd hh'
DAY_MINUTE | 'dd hh:mm'
DAY_SECOND | 'dd hh:mm:ss'
YEAR_MONTH | 'yy-mm'

Formatting Dates and Times

DATE_FORMAT(date, format) helps in representing dates and times in other formats than the usual MySQL format. Two examples illustrate the syntax:

DATE_FORMAT('2003-12-31', '%M %d %Y') returns 'December 31 2003'
 DATE_FORMAT('2003-12-31', '%D of %M') returns '31st of December'

Names of days of the week, months, etc., are always given in English, regardless of the MySQL language setting (language option).

Time Symbols in DATE_FORMAT, TIME_FORMAT, and FROM_UNIXTIME

%S, %s | seconds, two-digit | 00–59
%i | minutes, two-digit | 00–59
%k | hours (24-hour clock) | 0–23
%H | hours, two-digit, 0 to 23 o'clock | 00–23
%l | hours (12-hour clock) | 1–12
%h, %I | hours, two-digit, to 12 o'clock | 01–12
%T | 24-hour clock | 00:00:00 to 23:59:59
%r | 12-hour clock | 12:00:00 AM to 11:59:59 PM
%p | AM or PM | AM, PM

Date Symbols in DATE_FORMAT, TIME_FORMAT, FROM_UNIXTIME

%W | day of week | Monday to Sunday
%a | day of week abbreviated | Mon to Sun
%e | day of month | 1–31
%d | day of month two-digit | 01–31
%D | day of month with ending | 1st, 2nd, 3rd, 4th . . .
%w | day of week as number | 0 (Sunday)–6 (Saturday)
%j | day in year, three-digit | 001–366
%U | week number, two-digit (Sunday) | 00–52
%u | week number, two-digit (Monday) | 00–52
%M | name of month | January–December
%b | name of month abbreviated | Jan–Dec
%c | month number | 1–12
%m | month number, two-digit | 01–12
%Y | year, four-digit | 2002, 2003, . . .
%y | year, two-digit | 00, 01, . . .
%% | the symbol % | %

A few remarks about the week number are in order: %U returns 0 for the days from before the first Sunday in the year. From the first Sunday until the following Saturday, it returns 1, then 2, etc. With %u youget the same thing, with the first Sunday replaced by the first Monday.

GROUP BY Functions

The following functions can be used in SELECT queries (frequently in combination with GROUP BY ):

USE  mylibrary
SELECT  catName,  COUNT(titleID)  FROM  titles, categories
WHERE  titles.catID=categories.catID
GROUP BY catName
ORDER BY catName
catName | COUNT(titleID)
Children's books | 3
Computer books | 5
Databases | 2
. . . |

Since MySQL 4.1, the desired sort order can be specified in some aggregate functions, as in MAX(column COLLATE collname).

Aggregate Functions

AVG(expr) | computes the average of expr
BIT_AND(expr) | performs a bitwise AND of expr
BIT_OR(expr) | performs a bitwise OR of expr
COUNT(expr) | returns the number of expr
COUNT(DISTINCT expr) | returns the number of different expr expressions
MAX(expr) | returns the maximum of expr
MIN(expr) | returns the minimum of expr
STD(expr) | computes the standard deviation of expr
STDDEV(expr) | like STD(expr)
SUM(expr) | computes the sum of expr
VARIANCE(expr) | computes the variance of expr (since MySQL 4.1)

Additional Functions

Administrative Functions

BENCHMARK(n, expr) | executes expr a total of n times and measures the time elapsed
CONNECTION_ID( ) | returns the ID number of the current database connection
CURRENT_USER( ) | returns the name of the current user in the form in which authentication takes place (that is, with the IP number instead of the host name, e.g., "radha@127.0.0.1")
DATABASE( ) | returns the name of the current database
FOUND_ROWS( ) | returns since MySQL 4.0 the number of records found by a SELECT LIMIT query if in the SELECT command, the option SQL_CALC_FOUND_ROWS was used
GET_LOCK(name, time) | defines a lock with the name name for the time time (in seconds); usually executed with DO, as in DO GET_LOCK('abc', 10)
IDENTITY( ) | since MySQL 4.0 equivalent to LAST_INSERT_ID( )
IS_FREE_LOCK(name) | tests whether the lock name is available; returns 0 if the lock is currently in use (thus before GET_LOCK was executed), otherwise, 1
LAST_INSERT_ID( ) | returns the AUTO_INCREMENT number most recently generated within the current connection to the database
RELEASE_LOCK(name) | releases the lock name
SESSION_USER( ) | equivalent to USER( )
SYSTEM_USER( ) | equivalent to USER( )
USER( ) | returns the name of the current user and associated host name (e.g., "root@localhost")
VERSION( ) | returns the MySQL version number as a string

Miscellaneous

BIT_COUNT(x) | returns the number of set bits
COALESCE(list) | returns the first element of the list that is not NULL
LOAD_FILE(filename) | loads a file from the local file system

The two functions GET_LOCK and RELEASE_LOCK do not execute a lock; that is, neither MySQL nor a database is blocked. These functions serve, rather, for communication between processes. As long as a process has a lock with a particular name defined, no other process can obtain a lock with the same name. The second process in this case receives, with GET_LOCK, the return value 0 and therefore knows that currently another process has obtained a lock. (As a rule, the second process must just wait a bit and then try again.)


Chapter 19: MySQL Tools

This chapter is a reference for the options and functions of the most important MySQL tools. We discuss the server mysqld, the monitor mysql, and the administration tools mysqladmin, myisamchk, etc.

These tools have a number of common options, and they evaluate configuration files in the same manner. For this reason, the chapter begins with a section describing these common properties.

Overview

The common feature exhibited by the commands introduced in this section is that they are launched as external programs in a command window (Windows) or in a command shell (Unix/Linux). The entire operation of these programs is carried out in text mode and is therefore not what one would term excessively convenient. However, these commands are very well suited for execution in scripts in the automation of administrative tasks.

The following list provides an overview of the commands discussed in this section.

MySQL Server and Included Administration Tools

mysqld* | starts the MySQL server, known under Unix/Linux as the MySQL daemon (whence the letter "d"); under Windows, there are several variants (mysqld, mysqld-nt, mysqld-opt); see Chapter 2
mysqld_safe | should be used under Unix/Linux for a secure server launch
mysql | enables interactive execution of SQL commands
mysqladmin | assists in various administrative tasks (display status, reinput privileges, execute shutdown, etc.)
mysqldump | saves contents of a MySQL database in a text file
mysqlimport | inputs a table from a text file
mysqlshow | displays information on databases, tables, and columns
mysqlbug | sends (via e-mail) an error message (Unix/Linux only)
myisamchk | checks the integrity of MyISAM table files and repairs them as necessary
myisampack | compresses MyISAM table files for more efficient read-only access
Pointer

The two Perl scripts mysqlaccess and mysql_setpermission assist in the administration of MySQL access privileges. These scripts were described in Chapter 9, in the discussion of managing access privileges.

There are several additional scripts and programs for special tasks, and these were presented in Chapter 10 (e.g., mysqlhotcopy). However, some of these programs are available only under Unix/Linux.
Remark

Most of the commands presented here are compiled programs (*.exe under Windows). However, others are in script form and can be executed only if a suitable script interpreter is installed. This is usually the case under Unix/Linux. Under Windows, it may be that you have to install Perl yourself (see Chapter 2).


Common Options and Configuration Files

A common feature of the programs described in this chapter is that there are certain options that can be used by almost all of the commands, and these options can be preset in a common configuration file, so as to save typing when the commands are invoked.

Common Options

Various options can be passed to all commands when they are executed. As is usual with Unix/Linux, commands can be prefixed with a hyphen (short form) or two hyphens (full option name). Please note that the short forms of options are case-sensitive.

Common Options of mysql, mysqladmin, mysqld, mysqldump, mysqlimport, myisamchk, myisampack

--help | displays a brief operation introduction
--print-defaults | displays default values for options; default values can come from configuration files or system variables
--nodefaults | causes no configuration files to be read at startup
--defaults-file=filename | causes only this configuration file to be read at startup
--defaults-extra-file=filename | first the global configuration file is read, and then filename, and finally (only under Unix/Linux), the user-specific configuration file
--port=n | specifies the TCP/IP port over which communication takes place (usually 3306)
--socket=filename | specifies which socket file should be used for local communication between client and server (only under Unix/Linux; by default usually /var/lib/mysql/mysql.sock)
--version | displays the version number of the program

Common Options of the MySQL Client Tools

-u un | --user=username | determines the username for registration with MySQL
-p | --password | asks for input of password immediately after start of the command
-pxxx | --password=xxx | passes the password directly; in contrast to other options, there can be no space after -p; this is more convenient than interactive input of the password, but it can represent a considerable security risk and thus should generally be avoided; under some operating systems, any user can see the password by looking at the process list
h emphhn | --host=hostname | gives the name of the computer on which the server is running (assumed by default to be localhost, that is, the local computer)
-W | --pipe | uses a named pipe for communication with the MySQL server. Named pipes are available only for local connections under Windows NT/2000/XP and only if MySQL was appropriately compiled (which is not the default)
-C | --compress | minimizes the data flow between client and server by making use of data compression
Tip

For a connection to the MySQL server to be at all possible, the following two options must generally be used at the start of each client command:

> mysql -u username -p
Enter Password: xxxxxx

If MySQL is not yet password-secured, then this will work, of course, without a password being specified. Information on user and privilege management in MySQL can be found in Chapter 9.
Caution

If you execute MySQL commands under Windows and create a directory with options, then instead of the backslash you should use the forward slash (/). If the file name contains space characters, then put the entire path in quotation marks, as in the following example:

--character-sets-dir="Q:/Program Files/mysql/share/charsets"

Setting Options in Configuration Files

If you observe that you are using particular options over and over, you can save these in options files for many of the commands covered in this section. The options are used by mysql, mysqladmin, mysqld, mysqldump, mysqlimport, myisamchk, myisampack, and mysqld_safe.

The following list collects the locations where the options files must be stored. At startup, all options files—those that exist already—are read in the order in which they are listed below. In the case of contradictory settings, the most recently read options file takes precedence. (Whether and what configuration files will be read depends on the options --no-defaults, --defaults-file, and --defaults-extra-file; see above.)

Where Options Files Are Stored

Validity | Windows | Unix/Linux
global options user-specific options (no evaluation by mysqld) | C:\my.cnf, Windows\my.ini | /etc/my.cnf˜/.my.cnf
server-specifc options (for mysqld only) | DATADIR\my.cnf | DATADIR/my.cnf

The directory DATADIR is the default directory, which during compilation of MySQL is provided as data directory. Under Windows, this is normally C:\mysql\data, while under Unix/Linux, it is generally /var/lib/mysql. Note that DATADIR need not be the directory in which MySQL actually stores the database files. This directory is usually set when the MySQL server is launched with the option --datadir. However, access to the configuration file takes place before this option is evaluated.

The syntax of the file is based on the following pattern:

#  comment
[programname]
option1    #  equivalent to --option1
option2=abc   # equivalent to --option2=abc

These options are divided into groups for each program. Instead of programname, you should specify the name of the program:

  • Settings that are to be used by all programs other than mysqld are assigned to the group [client].

  • Settings that relate only to the server are assigned to the group [mysqld]. (The group [server] is also used by mysqld.)

  • Settings special to the program xyz are assigned to the group [xyz].

Here, option is the option name in long form, but without hyphens. (So, for example, the option --host in the configuration file becomes simply host.) If options expect parameters, then these are specified with =.

Remark

In MySQL 3.23, there was a distinction between options and variables in the configuration settings. Variables had to be set with set-variable varname=value. Beginning with MySQL 4.0, the intuitive short form varname=value is permitted, regardless of whether a configuration key word is considered an option or a variable. (The old form continues to be accepted.)

Let us see, finally, a concrete example of a configuration file:

# configuration file /etc/my.cnf or Windows\my.ini
# options for all MySQL tools
[client]
user=username
password=xxx
host=uranus.sol
# options for mysqldump
[mysqldump]
force
# options for mysql (monitor)
[mysql]
safe-updates
select_limit=100
Tip

Changes to configuration files are effective only after a restart of the program in question. This holds in particular for the MySQL server (thus for options in the group [mysqld]).
Caution

Please be sure that the options specified in the [client] section are truly supported by all MySQL tools. If a MySQL tool finds an unknown option in the [client] section, then the command is terminated with an error message.
Caution

If you wish under Unix/Linux to execute user-specific options settings in ~/.my.cnf and possibly specify passwords there, then ensure that no other users are able to read this file.

user$ chmod 600 ~/.my.cnf
Caution

If you specify Windows paths or directories in a configuration file, then you must use / or \\ instead of the backslash \. (Inthe Windows version of MySQL, the backslash is used as escape character.)

Furthermore, paths cannot be placed in quotation marks within a configuration file (even if the path contains space characters).

Please note that these rules are different from those that obtain for a direct setting of options with --option = ... .
Tip

Under Unix/Linux, you can use the program my_print_defaults grp to determine the options set for [grp] in the configuration file. The program is particularly suited for use in custom scripts.

Memory Specifications

For options and variables that expect a memory specification, the letters K, M, G may be used to denote kilobytes (1024 bytes), megabytes, and gigabytes. Thus the settings key_buffer_size=16M, key_buffer_size=16384K, and key_buffer_size=16777216 are equivalent.

Options in Environment Variables (aka System Variables)

An additional possibility for specifying options are environment variables at the level of the operating system. (Under Windows, these variables are usually known as system variables.)

The following list names the most important of these variables. To set such variables under Windows, you use the dialog for system control (see Figure 4-1). Under Linux you can define such variables in script files (e.g., in /etc/profile or ~/.profile) with export. Depending on which shell you use, you may use the command declare -x or setenv instead of export.

Import and Environment Variables for mysql, mysqladmin, mysqld, mysqldump, ...

MYSQL_TCP_PORT | specifies the port number for the TCP/IP connection to MySQL (generally 3306)
MYSQL_UNIX_PORT | specifies the socket file for local communication under Linux/Unix (e.g., /var/mysql.lock)
TMPDIR | specifies the directory to be used for temporary files; this directory is also used for temporary tables
USER | specifies the user name

Precedence

Option settings are read in the following order: environment variables, configuration files, options at program startup.

In the case of contradictory settings, the last setting read takes precedence. For example, options at program startup supersede settings in environment variables.

New in MySQL 4.0

There are many changes relating to options and configuration settings beginning with MySQL 4.0.2:

  • As already mentioned, variables can now be specified directly (thus --variable=123 instead of --setvariable variable=123). This holds as well for settings in configuration files.

  • A hyphen may replace an underscore in variable names (thus --variable-name=123 instead of --variable_name=123).

  • Options that activate a particular function were previously used simply in the form --option. Now, with many such options, several forms are permitted for activating or deactivating a function:

    --option          activates the function (as previously)
--option=1        activates the function
--enable-option   activates the function
--option=0        deactivates the function
--disable-option  deactivates the function
--skip-option     deactivates the function

    Note, however, that there are options like --skip-grant-tables for which skip is an integral part of the operator name. In this case, --enable-grant-tables and -grant-tables=0 do not function.

  • There are MySQL server variables that can be changed by the user at run time specifically for the currently active connection. To do this, the user simply executes an SQL command like SET read_buffer_size=16M. For such variables, an upper bound can be set at server start by prefixing --maximum,asin --maximum-read_buffer_size=32M.

  • Every new version of MySQL offers new options. If you are writing a script and do not wish to prescribe a particular version of MySQL, you may use the new option --loose as a prefix (e.g., --loose-optionname=3). If the option exists, it will be properly set. On the other hand, if the program does not recognize the option, then the setting is ignored without an error message.

Since the syntax of MySQL 3.23 is still allowed for compatibility, there are often various ways of making a particular configuration setting. This situation makes a complete and standardized documentation difficult.


mysqld (Server)

The following lists, organized by topic, summarize the most-used mysqld options.

Pointer

A complete reference to the long list of mysqld options can be obtained in mysqld --help. In the MySQL documentation there is unfortunately no unified description of all options. Instead, it is distributed among a number of sections (e.g., replication options, InnoDB options). The following addresses will lead you to the most important of these:

http://www.mysql.com/doc/en/Command-line_options.
http://www.mysql.com/doc/en/Replication_Options.html
http://www.mysql.com/doc/en/InnoDB_start.html

The MySQL server is generally started as a service (Windows NT/2000/XP) or via the script mysqld_safe (Unix/Linux). In such a case, it is impossible to specify options directly. Therefore, mysqld options are specified almost exclusively in configuration files. For this reason, in the following tables, the options are given in the manner in which they would appear in a configuration file (that is, without the prefixed --).

Twoexceptions are the options --defaults-extra-file and --user:

  • With --defaults-extra-file=filename, an additional configuration file can be specified that is read after all other configuration files.

  • user=name specifies the Unix/Linux account under which mysqld is to be executed. For a change of account to take place at the start of MySQL, mysqld must be started by root. Normally, mysqld is started by the script mysqld_safe with --user =mysql.

Both options must be passed directly to mysqld or to the start script mysqld_safe. They cannot be specified in a configuration file.

Basic Settings

mysqld Language Setting

default-characterset=name | specifies the character set to be used (for sorting, comparison, etc.)
language=name | specifies the language in which error messages, etc., are to be output

mysqld Directories and Files

basedir=path | uses the given directory as base directory (installation directory)
character-sets-dir=path | specifies the directory for the character set files
datadir=path | reads database files from the specified directory
pid-file= filename | specifies the file in which the process ID number should be stored (under Unix/Linux only); the file is evaluated by the init-V script to terminate mysqld
socket=filename | specifies the file name mysql.sock (only Unix/Linux, by default generally /var/lib/mysql/mysql.sock)
lower_case_table_names=1/0 | specifies whether in the creation of new directories and tables, lowercase should be used exclusively; under Windows, since MySQL 4.0.2 and 4.1.1 default is 1

mysqld Security, Network Access

enable-named-pipes | enables a connection under Windows NT/2000/XP be- tween client and server with named pipes; this option has effect only if the MySQL server was compiled to support named pipes
local-infile[=0] | activates/deactivates the ability to process local files with LOAD DATA LOCAL
myisam-recover[=opt1, opt2, . . .] | has the effect that at startup, all damaged MyISAM tables are automatically restored; the possible options are DEFAULT, BACKUP, QUICK, and FORCE; they correspond to the myisamchk options
old-passwords | passwords in the mysql database in MySQL 4.1 are encrypted as in MySQL 3.23 and 4.0 (by default, a new, more secure, encryption is used); this option is available since MySQL 4.1
safe-user-create | a user can create a new user with GRANT only if he has the Insert privilege for the table mysql.user; this is a supplementary security mechanism (the user also needs the Grant privilege to be able to execute GRANT
skip-grant-tables | omits input of mysql database with access information; (caution: anyone can change any database!)
skip-host-cache | a cache is not used to store the association between computer names and IP numbers
skip-name-resolve | suppresses the resolution of IP numbers into the corresponding host names, resulting in access control (table mysql.user) exclusively via IP numbers
skip-networking | permits only local connections over a socket file (Unix/Linux) or over named pipes (Windows), not over TCP/IP; this increases security but excludes external connections over a network and excludes all Java clients (which use TCP/IP even in a local connection)
user=name | specifies the Unix/Linux account under which mysqld is to be executed; for the switch to work after startup, mysqld must be started up from root; normally, mysqld is started via the script mysqld_safe with user=mysql

mysqld: Memory Usage, Tuning, Query Cache

bulk_insert_buffer_size=n | specifies how much memory is allocated for execut- ing INSERT commands in which many records are simultaneously inserted (default 8 megabytes)
key_buffer_size=n | specifies how much RAM is reserved for index blocks (default 8 megabytes).
join_buffer_size=n | specifies how much memory is to be allocated for JOIN operations when there is no index for the JOIN columns (default 128 kilobytes)
max_heap_table_size=n | specifies the maximum size of HEAP tables (default 16 megabytes); if this size is exceeded, the table is stored in a temporary file instead of in RAM
max_connections=n | specifies the maximum number of simultaneous database connections (default 100)
query_cache_limit=n | specifies the maximum size of a query result to be stored in the query cache (default 1 megabyte)
query_cache_size=n | specifies the size of the query cache (by default 0, meaning that the query cache is inactive)
query_cache_type=0/1/2 | specifies the mode in which query cache runs: 0: query cache is off 1: query cache is on (default mode) 2: demand mode only for SELECT SQL_CACHE
read_buffer_size=n | specifies how much memory is reserved for each thread for reading sequential data from tables (default 128 kilobytes); the parameter can be set as needed with SET SESSION read_buffer_size=n
read_rnd_buffer_size=n | similar to read_buffer_size, but holds for the case that the records are to be output in a particular order, e.g., with ORDER BY (default 256 kilobytes)
sort_buffer=n | specifies the size of the buffer for sorting (default 2 megabytes); if the buffer is too small, a temporary file must be used
table_cache=n | specifies the maximum number of open tables (default 64)
tmp_table_size=n | specifies the maximum size of temporary HEAP tables (default 32 megabytes); if this size is exceeded, the tables are transformed to MyISAM tables and stored in a temporary file

Logging, Replication

mysqld Logging

log[=file] | logs every connection as well as all SQL commands (general query log); if no file name is specified, then MySQL uses the file name hostname.log in the database directory
log-slow-queries[=file] | logs queries whose execution takes longer than the time specified in the variable long_query_time (slow query log)
long_query_time=n | specifies the time limit for slow queries (default 10 seconds)
log-long-format | logs not only slow queries, but also all queries that are carried out without the use of an index (holds for log-slow-queries)
log-bin[=filename] | logs all SQL commands that make a change in the data, in particular, all INSERT, UPDATE, and DELETE commands; a binary format is used (binary update log); as file name, file.n or the default hostname.n is used, where n is a three-digit integer (the logging files are sequentially numbered)
log-bin-index=filename | specifies the file name for the index file for binary logging
max_binlog_size=n | specifies the maximal size of a binary logging file (default 1 gigabyte)
binlog-do-db=dbname | limits binary logging to the specified database; changes in other databases are not logged; to specify more than one database, the option must be repeated in the configuration file, one database in each line
binlog-ignore-db=dbname | excludes the database from binary logging; changes in this database are not logged
log-update[=file] | has the same function as log-bin, but the logging is in text format (text update log); use the more efficient binary logging if you can
log-error=file | specifies the file name for the error logging file (error log); this logging variant cannot be deactivated; if the option is not specified, MySQL uses the file name hostname.err

mysqld Replication (Master)

server-id=n | assigns the server a unique ID number; n can be between 1 and 231
log-bin | activates binary logging
binlog-do/ignore-db=dbname | logs only the specified databases or ignores the specified databases in logging

mysqld Replication (Slave)

server-id=n log-slave-updates | assigns a unique ID number to the server executes logging on the slave so that the computer can be used to continue a replication chain (A B C)
master-host=hostname | specifies the host name or IP address of the replication master; this setting is ignored once replication is set up if the file master.info exists
master-user=replicusername | specifies the user name for replication communication; ignored once master.info exists
master-password=pword | specifies the associated password; ignored if master.info exists
master-port=n | specifies the port number of the master server (default 3306); ignored if master.info exists
master-connect-retry=n | specifies after how many seconds an attempt to reestablish a broken connection to the master can be made (default 60); ignored if master.info exists
master-ssl-xxx=xxx | enables the configuration of SSL communication between master and slave; according to the MySQL documentation (as of version 4.0.12), it has not been adequately tested
replicate-do-table = dbname.tablename | replicates only this table; to specify more than one table, the option must be repeated (each table requires its own line in the configuration file)
replicate-wild-do-table =dbname.tablename | functions like replicate-do-table except that the wild card % is allowed to form a pattern (e.g., test%.% to replicate all tables of all databases whose name begins with test)
replicate-do-db=dbname | replicates only this database
replicate-ignore-table = dbname.tablename | excludes this table from replication
replicate-wild-ignore-table = dbn.tablen | excludes these tables from replication
replicate-do-db=dbname | excludes this database from replication
replicate-rewrite-db =db1name->db2name | replicates the database db1name on the master under the name db2name on the slave
report-host=hostname | specifies the host name of the slave; this information is relevant only for the command SHOW SLAVE HOSTS, which allows on the master system the creation of a list of all slaves
slave_compressed_protocol=1 | enables compressed communication between master and slave if both servers are capable of it
slave-skip-errors=n1,n2, . . . or all | continues replication even if errors n1, n2, etc., have occurred, or (with all) regardless of any errors be- ing triggered; under a correct configuration, no errors should occur on the slave due to SQL com- mands being executed (if an error occurs on the master, the command is not replicated); without slave-skip-errors, replication is stopped by an error and must be manually resumed

InnoDB Configuration

mysqld InnoDB: General Settings, Tablespace Files

skip-innodb | specifies that the InnoDB table driver should not be loaded; useful to save memory when no InnoDB tables are to be used
innodb_data_home_dir=p | specifies the InnoDB directory; all additional directory or file specifications are relative to the path p; by default, the InnoDB driver uses the MySQL data directory
innodb_data_file_path=ts | specifies the tablespace for all InnoDB tables; can involve more than one file; the size of each file must be specified in bytes, megabytes (M), or gigabytes (G); the names of the tablespace files are separated by semi- colons; for the last of the tablespace files, the attribute autoextend and a maximal size (max:n) can be specified (e.g., ibdata1:1G;ibdata2:1G:autoextend:max:2G, by which ibdata1 has size 1 gigabyte, ibdata2 is set up with the same size and can grow as large as 2 giga- bytes); instead of a file, the device name of a hard-drive partition can be given; in this case, the key word newraw must follow the exact size for initialization, and for fur- ther use, the key word raw (e.g., /dev/hdb1:20Gnewraw or /dev/hdb1:20Graw); the default setting since MySQL 4.0 is ibdata1:10M:autoextend
innodb_lock_wait_timeout=n | specifies how long (in seconds) should be waited for the release of locks; after this time, the transaction is broken off with ROLLBACK ; the setting is espe- cially important when deadlocks develop that the InnoDB driver does not recognize; the default value is 50 seconds
innodb_fast_shutdown=On/Off | specifies whether InnoDB should shut down as quickly as possible; in the default setting On, InnoDB does not transfer the INSERT buffer into the tables; this process is repeated at the next start of the MySQL server (the setting On does not represent a risk, since the INSERT buffer is a component of the tablespace; thus no data can be lost; the setting Off is more dangerous, since in a computer shutdown it could happen that InnoDB would not have enough time for its synchronization work and would be stopped in its tracks by the operating system)

mysqld Options: InnoDB Tables

mysqld InnoDB: Logging

innodb_log_group_home_dir=lp | specifies the directory for InnoDB logging files (ib_logfile0, 1, etc.); by default, the InnoDB driver uses the MySQL data directory
innodb_log_files_in_group=n | specifies how many logging files should be used (default is 2); the InnoDB table driver fills these files sequentially; when all files are full, the first is overwritten with data, etc.
innodb_log_file_size=n | specifies how large each logging file should be (de- fault 5 megabytes); the size specification uses M for megabytes and G for gigabytes
innodb_flush_log_at_trx_commit=0/1/2 | determines when data should be written to the logging file and when these files should be synchronized (that is, when changes should be physically stored to the hard disk); the default setting 0 means that the data are written and synchronized about once per second; the setting 1 is more secure (save and synchronize after each COMMIT); a compromise is the setting 2, with writing after COMMIT and synchronization only about once per second
innodb_flush_method=x | under Unix/Linux, specifies how logging files are synchronized (i.e., how they are stored on the hard drive); choices are fdatasync: synchronization with fsync( ) (default) and O_DSYNC: update with O_SYNC( )
innodb_log_archive=1 | activates archive logging in the files ib_arch_log_n; this type of logging does not make sense when InnoDB is used together with MySQL (instead, use binary logging of the MySQL server); archive logging is therefore by default deactivated immediately after the start of the MySQL server (only the creation of the new tablespace or transaction logging files are logged)

mysqld InnoDB: Buffer Settings and Tuning

innodb_buffer_pool_size=n | specifies how much RAM should be used for table data and indexes (default 8 megabytes); this parameter has considerable influence on speed and should be up to 80% of available RAM on computers used exclusively as MySQL/InnoDB database servers
innodb_log_buffer_size=n | specifies how much RAM should be used for writing the transaction logging files (default 1 megabyte)
innodb_additional_mem_pool_size=n | specifies how much RAM should be used for various internal management structures (default 1 megabyte)
innodb_file_io_threads=n | specifies how many threads for I/O operations, that is, for access to the hard drive, should be used (default 4)
innodb_thread_concurrency=n | specifies how many threads InnoDB should use altogether (default 8)

mysqld Miscellaneous

bind-address=ipadr | specifies the IP address that MySQL should use; this option is important if the computer is so configured that it uses several IP addresses
default-table-type=type | specifies the table type that new tables are to use if the type is not explicitly given (default is MyISAM)
ft_min_word_len=n | specifies the minimal word length for a full-text index; in the default setting 4, words with three or fewer letters are not included
max-allowed-packet=n | specifies the maximum packet size for data exchange between client and server; max-allowed-packet must be at least as large as the largest BLOB to be processed by client programs; default is 1 megabyte
new | makes MySQL 4.0 behave is some ways like MySQL 4.1; this enables uncomplicated testing of many changes; at the moment, all that is affected is the representa- tion of TIMESTAMPs in the new form YYYY-MM-DD HH:MM:DD
port=n | specifies the IP port that the server should use (default 3306)
Caution

If you specify an option in a configuration file that mysqld does not know about (for example, due to a silly typo), then the server cannot start up. So watch out!


mysqld_safe (Server Start)

Under Unix/Linux, the MySQL server is usually started by an Init-V script (e.g., /etc/init.d/mysql or /etc/init.d/mysql). This script is automatically executed at the startup of the operating system. It does not directly launch the server, but instead, executes the script mysqld_safe. It is this script that starts the MySQL server. (In MySQL 3.23, the script is called safe_mysqld.)

The script mysqld_safe is started with root privileges. The MySQL server, on the other hand, is started with the privileges of a user designed for this purpose (normally with the name mysql).

After the start of the MySQL server, mysqld_safe continues to run until it is explicitly ended (e.g., with mysqladmin shutdown). If the MySQL server is ended unexpectedly due to a crash, then it is immediately restarted by mysqld_safe.

Usually, only a few options are passed to mysqld_safe (e.g., --data-dir and --pid-file), which transmit the script to the MySQL server mysqld. There are also some options specifically for mysqld_safe.

mysqld_safe Options

--core-file-size=n | limits the size of the core file, which contains an image of the server after a crash and makes debugging possible; n must be given in bytes
--ledir=dirname | specifies the directory in which the MySQL server is located (by default /usr/sbin)
--mysqld=filename | specifies the name of the MySQL server that is to be started (default mysqld)
--mysqld-version=suffix | specifies an extension to the server name (e.g., max if mysqld-max should be started instead of mysqld)


mysql_install_db (New Installation of mysql Database)

Under Unix/Linux, the mysql database can be freshly installed with the script mysql_install_db. For this, the server must be stopped. (The script starts the server itself with the option --bootstrap.)

Among other things, the script must be executed in the same account as is used for executing the MySQL server (usually mysql). You can also execute mysql_install_db as root,in which case you must subsequently change the owner of the database directory mysql and the files contained therein:

root#mysql_install_db root# chown mysql -R /var/lib/mysql/mysql

The script is generally executed automatically as a component of a new MySQL installation. The script is also suited to recreating the default state of the mysql database.

Caution

In a test, the script used uranus as the local host name, although the complete computer name is uranus.sol. The use of an abbreviated computer name can lead to login problems. It may be necessary to place the complete computer name manually in the mysql.user table (see also Chapter 9).

Although mysql_install_db is available under Windows (directory scripts), execution of it fails for the lack of a suitable shell interpreter.


mysql_fix_privileges (Updating the mysql Database)

The mysql database for managing MySQL access privileges is being continually extended and improved. If you carry out a MySQL update and use the mysql database from a previous installation (see Chapter 10), you should execute the script mysql_fix_privileges, which updates the database, inserts new columns as necessary, etc. The single parameter that must be passed to the script is the MySQL root password.

After the script has been executed, there is generally some work that has to be done by hand, because the script does not always know whether certain new privileges should be set. In case of doubt, the script elects not to set, which is more secure, but for some MySQL users, it reduces their privileges with respect to previous versions of MySQL.

Although the script is available under Windows (directory scripts) as well as under Unix/Linux, execution under Windows fails for the lack of a suitable shell interpreter.


mysql_fix_extensions (Renaming MyISAM Files)

In transferring database directories with MyISAM tables from Windows to Unix/Linux, the problem sometimes occurs that the case (upper/lower) of the file identifier is incorrect. (Unix/Linux is case-sensitive, while Windows could use some case-sensitivity training.) MySQL expects, for example, *.frm,but *.MYI and *.MYD.

The Perl script mysql_fix_extensions solves this problem. For some strange reason, in MySQL 4.0, the script is included only with the Windows version of MySQL (directory scripts), although it is usually used under Unix/Linux after table files have been copied onto a Unix/Linux computer. (With MySQL 4.1, the script is also installed with the Linux version.)

The single parameter passed to the script is the name of the MySQL data directory. There are no options.


mysql (SQL Command Interpreter)

The monitor mysql allows interactive execution of SQL commands. This program can also be run in batch mode for many administrative tasks, including the generation of HTML tables. The commands are input from a file with < file. All SQL commands can be used in this file. The commands must be followed by a semicolon. Comments are introduced with the character #. (All additional characters to the end of the line are then ignored.)

At the launch of mysql, numerous options can be specified. Furthermore, an optional database name can be given, in which case this database becomes the default database (corresponds to the command USE databasename).

mysql: Syntax

mysql [options] [databasename] [ < commands.sql]

mysql: General Options

-e cmd | --execute=cmd | executes the given command(s); commands must be separated by semicolons and placed in quotation marks;
-i | --ignore-space | recognizes functions even if there are spaces between function names and their parameters; with -i, for example, SUM (price) is allowed, while without the option it must be written SUM(price)
-L | --skip-line-numbers | displays error messages without line numbers; the line numbers otherwise displayed refer to the location at which the faulty SQL command is located in a batch file and is generally an aid in debugging
-U | --i-am-a-dummy | permits UPDATE and DELETE commands only
 | or --safe-updates | if the effective range is limited with WHERE or LIMIT; furthermore, there is a maximum number of query results from SELECT commands as well as a maximum number of JOINs
-V | --version | displays the version of mysql; mysql is then terminated
 | --tee=filename | copies all input and output into the specified logging file; this option is allowed only if mysql is used in interactive mode (not in batch mode)
 | --no-tee | does not use logging (the default)

mysql: Formatting and Output Options

-B | --batch | separates columns in tables by tab characters (instead of by spaces and lines); moreover, only results of queries are displayed, and no status information
-E | --vertical | lists the results of queries with columns displayed horizontally, one below the next (instead of with ver- tical columns, one next to the other); this option is particularly to be recommended if a query returns many columns but few rows (ideally only one); as with --batch, only results of queries are shown, and no status information
-H | --html | formats the results of queries as HTML tables; as with --batch, only results of queries are shown, and no status information
-N | --skip-column-names | leaves off column titles in the output of tables
-r | --raw | in query results, outputs the characters zero-byte, tab, newline, and \ unchanged (normally, these characters are output as \0, \t, \n, and \\); this option is effective only in combination with --batch
-s | --silent | displays less status information than in normal mode; does not use costly table formatting
-t | --table | formats tables with lines and spaces (default setting)
-v | --verbose | displays extensive status information, more than in normal mode
-X | --xml | formats the results of queries as an XML document

mysql: Commands for Interactive Mode

\c | clear | interrupts input of a command; \c can be given at the end of a command and leads to the entire input being simply ignored
\e | edit | calls the external editor named in the environment variable EDITOR and there enables a change in the command; this works only under Unix/Linux; after the return to mysql, the command given in the editor is not displayed in mysql, which makes this option somewhat confusing to use
\g | go | executes the command (equivalent to ; and Return)
\h | help | displays a list of commands
\p | print | displays the entire current command on the screen
\q | exit or quit | terminates mysql (under Unix/Linux, this works also with Ctrl+D)
\r | connect | terminates the current connection to MySQL and creates a new connection; optionally, a database name and host name of the MySQL server can be given (in that order)
\s | status | displays status information about the MySQL server
\T [fn] | tee [filename] | logs all input and output into the specified file; if no file name is given, then the file name used in the previous tee command is used; if the file already exists, then the input and output are appended to the end of the file
\t | notee | ends tee; logging can be resumed at any time with tee or \T
\u db | use database | makes the given database the default database
\# | rehash | creates (only under Unix/Linux) an internal list of all mysql commands, the most important SQL key words, and all table and column names of the current database; in the sequel, the input of the initial letters suffices; with Tab the abbreviation is extended to the full key word
\. fn | source filename | executes the SQL commands contained in the file; the commands must be separated by semicolons


mysqladmin (Administration)

Various administrative tasks can be accomplished with mysqladmin, such as creating new databases and changing passwords. There are several commands that can be passed to mysqladmin, which are then executed sequentially.

Tip

The names of mysqladmin commands can be abbreviated to the point where the name remains unique (e.g., flush-l instead of flush-logs,or k instead of kill).

Most mysqladmin commands can also be executed as SQL commands, for example, by CREATE DATABASE, DROP DATABASE, FLUSH, KILL, SHOW. These commands are given in parentheses in the lists below. There is additional information in the SQL reference in Chapter 18.

mysqladmin: Syntax

mysqladmin [options] command1 command2 . . .

mysqladmin: Options

-f | --force | no warnings are displayed (e.g., with drop database); further commands are executed even after errors
-i n | --sleep=n | repeats the command every n seconds (for ex- ample, for regular display of status or for ping); mysqladmin now runs endlessly; under Unix/Linux, it can be terminated with Ctrl + C, under Windows, only by closing the command window
-r | --relative | displays, in combination with -i and the com- mand extended-status, a change in the previous status
-E | --vertical | has the same effect as --relative, but changes are displayed in a single, very long, line
-t n | --timeout=n | sets the timeout time for establishing a connection to the server; if after n seconds no connection has been established, then mysqladmin is terminated
-w n | --wait=n | attempts n times to establish a connection to the MySQL server

mysqladmin: Commands

create dbname | generates a new database (corresponds to CREATE DATABASE)
drop dbname | deletes an existing database irrevocably (corresponds to DROP DATABASE)
extended-status | displays countless status variables of the server (SHOW STATUS)
flush-hosts | empties the host cache table (FLUSH HOSTS)
flush-logs | closes all logging files and then reopens them (FLUSH LOGS); with update logs, a new file is created, where the number of the file terminator is increased by 1 (e.g., file.003 file.004)
flush-status | resets many status variables to 0 (FLUSH STATUS)
flush-tables | closes all open tables (FLUSH TABLES)
flush-threads | empties the thread cache
flush-privileges | reinputs the privileges database mysql (FLUSH PRIVILEGES)
kill id1, id2, . . . | terminates the specified threads (KILL)
password newpassw | changes the password of the current user; under Windows, this command often causes problems; one could use instead the SQL command SET PASSWORD (see Chapter 9)
ping | tests whether a connection to the server can be established
processlist | displays all threads (SHOW THREADS)
reload | reinputs the privileges database mysql
refresh | closes all tables and log files and then reopens them
shutdown | terminates the SQL server
start-slave / stop-slave | starts/stops a slave process for replication
status | displays some server status variables
variables | displays the system variables of the SQL server (SHOW VARIABLES)
version | determines the version of the MySQL server


mysqldump (Backup/Export)

With mysqldump,you get a long list of all SQL commands that are necessary to re-create a database exactly as it was. There are three syntax variants of mysqldump, depending on whether a database, several enumerated databases, or all databases managed by MySQL are to be stored. Only with the first variant can the output be limited to particular tables.

mysqldump: Syntax

mysqldump [options] dbname [tables]

mysqldump [options] --databases [moreoptions] dbname1 [dbname2 . . . ]

mysqldump [options] --all-databases [moreoptions]

mysqldump: Options

 | --add-drop-table | inserts a DROP TABLE command before every CREATE TABLE; when tables are read in, existing tables are deleted
 | --add-locks | inserts LOCK TABLE before the first INSERT command and UNLOCK after the last IN- SERT command; generally speeds up reading in a database; the option should not be used with InnoDB tables
 | --all | specifies all MySQL-specific options in the CREATE TABLE command
-A | --all-databases | saves all databases managed by MySQL; CREATE DATABASE and USE are placed in the backup file
-B | --databases | stores several databases
 | --complete-inserts | generates for each data record a separate IN- SERT command; this is the default setting; use --extended-insert to minimize the size of the backup file
 | --delayed-inserts | creates INSERT commands with the option DELAYED
-K | --disable-keys | inserts ALTER TABLE . . . DISABLE KEYS or . . . ENABLE KEYS before or after INSERT commands; indexes are thereby not updated until the end of the insertion process, which is faster
-e | --extended-insert | generates few INSERT commands with which several records can be inserted simultaneously (more efficient and reduces the size of the backup file)
-F | --flush-logs | updates the logging files before the backup is begun
-f | --force | continues even after errors
 | --no-create-db | does not generate CREATE TABLE commands (only INSERTs)
 | --no-data | generates no INSERT commands (only CREATE TABLE commands to re-create the structure of the database)
-l | --lock-tables | executes a LOCK TABLE READ for all tables before the data are read; this ensures that no data can be changed while mysqldump is running; the obvious drawback is that all write processes are blocked until mysqldump is done, which can take a while with large databases (with InnoDB tables, you should use --single-transaction instead of --lock-tables)
 | --opt | shorthand for the following options: --add-drop-table, --add-locks --quick --all --extended-insert --lock-tables (in most cases, this is an optimal setting, though not when you use InnoDB tables, on account of --lock-tables)
-q | --quick | outputs results record by record without internal intermediate storage; without this option, first the entire table is moved into RAM and then output; the advantage of --quick is the lower memory requirement; the disadvantage is that the MySQL server is generally blocked for a longer period of time; --quick should definitely be used for very large tables (when the entire table cannot be held in RAM of the local computer)
-Q | --quote-names | encloses table and column names in single quotes (e.g., 'name')
 | --single-transaction | results in all tables being read within a single trans- action; this makes sense only when InnoDB tables are used, in which case this option ensures that no data are changed during output (with MyISAM tables, you should instead use --lock-tables)
-T dir | --tab=dir | writes the result directly into the specified direc- tory, whereby for each table two files are created, one with the table structure (*.sql) and the sec- ond with the stored data in the format of the command SELECT . . . INTO OUTFILE (*.txt)
-w cnd | --where=condition | considers only data records that satisfy the WHERE condition cnd or condition; the entire option must be placed in quotation marks, e.g., " -wprice>5 " or " --where=ID=3 "
-X | --xml | creates an XML file with the contents of the table (without information on the table structure)

mysqldump: Formatting Options (only in combination with --tab)

--fields-terminated-by | see Chapter 18: SELECT . . . INTO OUTFILE
--fields-enclosed-by |
--fields-optionally-enclosed-by |
--fields-escaped-by |
--lines-terminated-by |

If you are using --tab, then the second file (tablename.txt) contains the contents of the table directly (that is, not in the form of INSERT commands). This has several advantages: The resulting file is somewhat more compact, and a later importation can be executed significantly more quickly. (However, the operation is more complex, and only a single table can be handled.)

The options --fields and --lines should each be set in quotation marks. The following example shows how you can pass the double quote itself as a character to the option:

> mysqldump -u root -p --tab /tmp "--fields-enclosed-by=\"" ...

To reinput the file thus generated (*.txt) with mysqldump, you can use either the program mysqlimport, discussed in the following section, or the SQL command LOAD DATA.


mysqlimport (Text Import, Bulk Import)

With mysqlimport, it is possible to import specially formatted text files into MySQL tables. Here, mysqlimport represents merely an interface to the command LOAD DATA, described in detail in Chapter 10.

mysqlimport: Syntax

mysqlimport [options] databasename filename

mysqlimport: Options

-d | --delete | deletes all existing records in the table before importation
-i | --ignore | ignores new records with an existing value for a UNIQUE or PRIMARY KEY column
-L | --local | reads the file from the local file system (not from that of the MySQL server)
-l | --lock-tables | blocks the tables for all other clients during importation
-r | --replace | overwrites existing records with the same value in a UNIQUE or PRIMARY KEY column
 | --fields-terminated-by --fields-enclosed-by --fields-optionally-enclosed-by --fields-escaped-by --lines-terminated-by | affects special characters; see LOAD DATA in Chapter 18


mysqlshow (Displaying Information)

With mysqlshow, you can quickly obtain an overview of the databases, tables, and columns managed by MySQL. Without parameters, the command returns a list of all databases managed by MySQL. With parameters, the command displays information on the specified database, table, or column.

mysqlshow: Syntax

mysqlshow [options] [databasename [tablename [columnname]]]

mysqlshow: Options

-i

--status

displays additional status for tables (table type, average record length, etc.)


mysqlbug (Sending Error Notification)

The script mysqlbug helps in composing error messages to mysql@lists.mysql.com. The script is available only under Unix/Linux. It collects the most important information about the MySQL version installed on the system (version number of the server, options used at compilation, version number of the libraries, version number of Perl, etc.) and then displays a text template using the editor specified by $VISUAL.

You can now report information on your problems or errors at the locations provided for. As soon as you leave the editor, you have the possibility of terminating the error report, sending it, or editing it.

Tip

Under Windows, you can create error reports with the dialog sheet REPORT of the program WinMySQLadmin, instead of with mysqlbug.

Errors can also be reported in the error database of the MySQL web site:

http://bugs.mysql.com/


myisamchk (Repairing MyISAM Files)

With myisamchk you can check the integrity of MyISAM database files (name.MYD and name.MYI). Damaged files and indexes can be repaired as required. Re-creation of the index is necessary as well if the sort order of the server was changed.

The parameters of myisamchk are tables names. Here the complete file name is given (either without ending or with the ending *.MYI). Depending on the options specified, however, both MyISAM files, that is, name.MYD and name.MYI, are analyzed or changed.

Please note that with myisamchk there are several options whose significance depends on whether the program is used simply for checking (without -r, -o, --recover, --saferecover)or for changing table files (with one of these options).

If you are still working with ISAM tables (usually with older versions of MySQL), you can repair their files with isamchk. However, the options here are somewhat different from those of myisamchk.

myisamchk: Syntax

myisamchk [options] tablename1 tablename2 . . .

myisamchk: Options (Analyze Table File without -r or -o)

-c | --check | checks the integrity of the table files; -c is the default option if no options are specified
-e | --extend-check | checks most thoroughly (and slowly)
-F | --fast | checks only tables whose files were not properly closed
-C | --check-only-changed | checks only tables that were changed since the last check
-f | --force | restarts myisamchk with the option -r if errors are discovered
-i | --information | displays statistical information about the tables
-m | --medium-check | checks more thoroughly (and slowly) than with -c
-U | --update-state | marks a file as damaged if errors are discovered
-T | --read-only | does not change the file

myisamchk: Options (Repair and Change Table File with -r or -o)

-B | --backup | creates the backup file name.bak for name.myi
-e | --extend-check | attempts to re-create every data record; how- ever, this usually leads to many records with incorrect or deleted data; moreover, the re- pair takes a long time; this option should be used seldom
-f | --force | overwrites temporary files
-l | --no-symlinks | follows no symbolic links (that is, only those files are repaired that are found un- der the actual file name; available only under Unix/Linux)
-o | --safe-recover | like -r, but a different algorithm is used
-q | --quick | only an index file is repaired; the actual file is left untouched
-q -q |  | has almost the same effect as -q, but the data file is unchanged if key fields are not unique
-r | --recover | attempts to re-create defective files
-t p | --tmpdir=path | uses the specified directory for temporary files
-u | --unpack | decompresses table files that were compressed with myisampack
 | --character-sets-dir=dir | reads character set data from the specified directory
 | --set-character-set=name | uses the specified character set for recreating indexes

myisamchk: Other Options

-a | --analyze | analyzes and stores the distribution of key fields in the indexes; this can speed up table access somewhat
-A n | --set-auto-increment[=n] | uses as start value for AUTO-INCREMENT a number that is one greater than the highest value used so far, or else n (whichever is greater)
-d | --description | displays various information about the ta- ble (record format and length, character set, indexes, etc.)
-R | --sort-records=idxnr | sorts the records in the table file according to the specified index; the index is specified as a number (where this number can be deter- mined first with -d); then nearby records in the index are located near each other in the table file as well; this can speed up access if records are frequently read in the order defined by the index
-S | --sort-index | sorts the blocks of the index file

myisamchk: Variables for Memory Management

-O key_buffer_size=n | specifies the size of the key buffer (default 512 K)
-O read_buffer_size=n | specifies the size of the read buffer (default 256 K)
-O sort_buffer_size=n | specifies the size of the sort buffer (default 2 MB)
-O write_buffer_size=n | specifies the size of the write buffer (default 256 K)


myisampack (Compressing MyISAM Files)

MyISAM database files (name.MYD) are compressed with myisampack. Thereby one can achieve drastic reductions in storage requirements for tables (often considerably more than one-half) and under some circumstances increased access speed to the data. However, the data can now no longer be changed.

myisampack: Syntax

myisampack [options] tablename1 tablename2 . . .

myisampack: Options

-b | --backup | creates a backup name.old of the table file
-f | --force | executes the operation even if the resulting file is bigger than the original
-j | --join='new_table_name' | unites all tables specified into a single, large, file; the tables must all have exactly the same column definitions
-t | --test | executes myisampack provisionally without actually changing any data
-T p | --tmpdir=path | uses the specified directory for temporary files


Chapter 20: API Reference

This chapter contains a reference to the APIs (application interfaces) that were introduced in this book for the programming languages PHP, Perl, Java, and C.

PHP API

This reference gives a compact overview of the PHP functions and their parameters for access to MySQL databases. First a few formal remarks:

  • Square brackets in the left column indicate optional parameters.

  • For all functions for which there is an enumeration over an index n, this index is in the range 0 to nmax − 1.

  • Examples of the application of these functions can be found particularly in Chapters 3 and 11 through 13. Further references can be found in the Index.

Most of the functions presented here have been available since PHP 3, while some appeared in the course of development of PHP 4.n. All functions that came after PHP 4.0 are explicitly so designated.

Connection and Administration

$id =mysql_connect($host, $user, $pw); | establishes a connection
$id = mysql_connect($host, $user, $pw, $new_link, $client_flags); | as above, except that new_link specifies whether a new connection should be made if a similar connection already exists (de- fault false); client_flags specifies whether particular connection properties should be used (e.g., MYSQL_CLIENT_COMPRESS); these two optional parameters have been available only since PHP 4.3
$id=mysql_pconnect($host, $user, $pw); [, $new_link [, $client_flags]]); | establishes a persistent connection or attempts to reuse a still open connection of another PHP page
mysql_change_user($newuser, $passw); | changes the user name for the connection
mysql_select_db($dbname); | determines the default database
mysql_close([$id]); | closes the connection

Generally, the specification id can be omitted as long as there is only one connection to MySQL and thus no possibility of confusion.

Administration

$result = mysql_list_dbs([$id]); | determines a list of all known databases; the evaluation is like that of SELECT queries
$result = mysql_list_tables($dbname [,$id]); | determines a list of all tables of the database; evaluation like that of SELECT queries
$result = mysql_list_fields($dbn, $tbln [,$id]); | determines a list of all fields of the table; evaluation like that of SELECT queries
mysql_create_db($dbname [,$id]); | creates a new database
mysql_drop_db($dbname [,$id]); | deletes a database

Error Evaluation

$n = mysql_errno([$id]); | determines the number of the most recent error
$txt = mysql_error([$id]); | determines the error message

Information Functions

$txt = mysql_client_encoding([$id]); | since PHP 4.3 returns the name of the MySQL character set (e.g., "latin1")
$txt = mysql_get_client_info([$id]); | since PHP 4.0.5 returns a character string with the version number of the client library
$txt = mysql_get_host_info([$id]); | since PHP 4.0.5 returns a character string that describes the connection with the server (including host name, e.g., "localhost via TCP/IP")
$n = mysql_get_proto_info([$id]); | since PHP 4.0.5 returns an integer with the number of the communication protocol in use (e.g., 10)
$txt = mysql_get_server_info([$id]); | since PHP 4.0.5 returns a character string with the version number of the server (e.g., "4.0.12-nt")
$n = mysql_thread_id([$id]); | since PHP 4.3 returns an integer with the thread number of the given connection
$txt = mysql_stat([$id]); | since PHP 4.3 returns a character string with a brief status report on the server (e.g., "Uptime: 24763 Threads: 1 Questions: 65 . . . ")

Executing SQL Commands

[$result = ] mysql_query($sql [, $id]) | executes an SQL command for the default database; if it is a SELECT command, the found records can be evaluated with $result
[$result = ] mysql_db_query( $db, $sql [, $id]); | executes a command for the database db (which becomes the default database for all further queries)
$result = mysql_unbuffered_query( $sql [, $id]); | functions in principle like mysql_query, but is de- signed only for SELECT queries; the difference between this and mysql_query is that found records remain at first on the server and are transferred only as needed; the number of found records can be determined only by running through all of them; mysql_num_rows cannot be used
$sql = addslashes($s); | replaces zero-bytes and the characters ', ", and \ in $s with the strings \0, \', \", and \\
$sql = mysql_escape_string($s); | since PHP 4.0.3 functions like addslashes, but also replaces carriage return, line feed, and CTRL Z with the strings \n, \r, and \z
$sql = mysql_real_escape_string( $s [,$id]); | since PHP 4.3 functions like mysql_escape_string, but also considers the character set of the MySQL connection

Output of SELECT Query Results

mysql_data_seek($result, $rownr); | determines the active data record within the result
$row = mysql_fetch_array($result); | returns the next record of the result (or false); ac- cess to individual fields takes place with row[n] or row[' fieldname'], where case sensitivity is in force
$row = mysql_fetch_assoc($result); | functions like mysql_fetch_array, except that field access must be by column name; row[n] is not permitted
$row = mysql_fetch_row($result); | returns the next record of the result (or false); access to individual fields is via row[n]
$row = mysql_fetch_object($result); | returns the next record of the result (or false); access to individual fields is via row->fieldname
$data = mysql_result($result, $rownr, $colnr); | returns the contents of the field in row rownr and column colnr; this function is slower than the other functions in this list and should therefore be used only in particular cases (such as to read a single value, e.g., SELECT COUNT(*))
mysql_free_result($result); | frees the query result immediately (otherwise, not till the end of the script)

Here result is a value that enables access to the list of data records from a SELECT query. In evaluation by the functions in the following list there is always one record in the list that is active. Usually, all records are output one after the other with mysql_fetch_array, mysql_fetch_row, or mysql_fetch_object. The next record then automatically becomes the active one. The active record can also be set with mysql_data_seek

The three functions mysql_fetch_array, mysql_fetch_row, and mysql_fetch_object differ only in the way in which individual fields of a record are accessed: row['fieldname'], row[n], or row->fieldname. Of the three functions, mysql_fetch_row is the most efficient, but the difference in speed is very small.

Metainformation on Query Results

$n = mysql_num_rows($result); | determines the number of result records (SELECT)
$n = mysql_num_fields($result); | determines the number of result columns (SELECT)
$n = mysql_affected_rows([$id]); | determines the number of records that were changed by the last SQL command (INSERT, UPDATE, DELETE, CREATE, . . . , SELECT)
$autoid = mysql_insert_id([$id]); | determines the AUTO_INCREMENT value generated by the last INSERT command
$txt = mysql_info([$id]); | since PHP 4.3 returns status information on the last command, e.g., "Rows matched: 65 Changed: 65 Warnings: 0"; mysql_info is designed only for commands that usually affect large numbers of records (INSERT INTO, UPDATE, ALTER TABLE, etc.)

Metainformation on the Fields (Columns) of Query Results

$fname = mysql_field_name($result, $n); | returns the field name of column n
$tblname = mysql_field_table($result, $n); | returns the table name for column n
$typename = mysql_field_type($result, $n); | returns the data type of column n (e.g., "TINYINT")
$length = mysql_field_len($result, $n); | returns the maximum length of the column
$lengths = mysql_fetch_length($result); | returns a field with length information for all fields of the last-read data record (access with lengths[n])
$flags = mysql_field_flags($result, $n); | returns the attribute properties of a column as a character string (e.g., "not_null primary_key"); the properties are separated by spaces; evaluation is done most easily with explode
$info = mysql_fetch_field($result, $n); | returns information on column n as an object; evaluation proceeds with info->name (see the list below); note that info may contain, in part, properties other than flags

Attributes of mysql_field_flags

auto_increment | attribute AUTO_INCREMENT
binary | attribute BINARY
blob | data type BLOB, TINYBLOB, etc.
enum | data type ENUM
multiple_key | field is part of a nonunique index
not_null | attribute NOT NULL
primary_key | attribute PRIMARY KEY
timestamp | attribute TIMESTAMP
unique_key | attribute UNIQUE
unsigned | attribute UNSIGNED
zerofill | attribute ZEROFILL

Field Information for mysql_fetch_field

info->name | column name (field name)
info->table | name of the table from which the field comes
info->max_length | maximum length of the field
info->type | name of the data type of the field (e.g., "TINYINT")
info->numeric | 1 or 0, depending on whether the field contains numeric data
info->blob, not_null, multiple_key, primary_key, unique_key, unsigned, zerofill | 1 or 0; see list above for interpretation


Perl DBI

Remark

lease note that this reference does not contain an exhaustive list of all DBI methods, functions, and attributes. We have included only those key words that are most relevant for everyday MySQL programming with Perl. A complete reference can be found in the perldoc documentation.

In this book we generally place Perl methods in parentheses to improve readability. However, Perl syntax allows the execution of methods without parentheses, as in $dbh->disconnect.

The following lines shown the principles for building a Perl script file:

#!/usr/bin/perl -w
use DBI;                              # database access
use CGI qw(:standard);                # necessary only with CGI scripts
use CGI::Carp qw(fatalsToBrowser);    # only with CGI scripts
use HTML::Entities;                   # only with CGI scripts
. . .                                 # the actual code follows

Common Variable Names

The Perl DBI module is object-oriented. Thus the key words introduced in this section relate in part to methods that can be applied to specific objects (which in Perl are generally called Handles). In this reference the following variable names will be used for such objects:

Common Variable Names for DBI Handles

$dbh | (database handle) | represents the connection to the database
$sth | (statement handle) | enables evaluation of query results (with SELECT queries)
$h | (handle) | general handles, used in this section with methods that are available to $dbh, $sth, and DBI
$drh | (driver handle) | enables access to many administrative functions

Establishing the Connection

The Connection

use DBI( ); | activates the DBI module
$datasource = "DBI:mysql:dbname;" . | specifies database names and computer
"host=hostname"; | names; the database name may be omitted, but then at least the colon must be given
$dbh = DBI->connect($datasource, $username, $password [, %attributes]); | creates the connection to the database

Within the datasource character string, further parameters—separated by semicolons—may be given. (Information on setting up MySQL configuration files can be found in Chapter 19.)

Optional Parameters in the datasource Character String

host=hostname | specifies the name of the computer with the MySQL server (default localhost)
port=n | specifies the IP port (default 3306)
mysql_compression=0/1 | compresses communication (default 0)
mysql_read_default_file=filename | specifies the file name of the MySQL configuration file
mysql_read_default_group=mygroup | reads the group [mygroup] within the configuration (default group [client])

A list with attributes can be passed as an optional fourth parameter of connect. You can supply these attributes either directly or in the form of an array variable:

$dbh = DBI->connect($source, $user, $pw, Attr1=>val1, Attr2=>val2);
%attr = (Attr1=>val1, Attr2=>val2);
$dbh = DBI->connect($source, $user, $pw, \%attr);

To a great extent, these attributes can be read and changed with $dbh after the connection has been established:

$dbh->'LongReadLen' = 1000000;

The following table describes the most important connect attributes.

Optional connect Attributes ($dbh Attributes)

RaiseError=>0/1 | displays an error message and ends the program if the connection is not properly established (default 0)
PrintError=>0/1 | displays an error message but continues execution if the connection is not properly established (default 1)
LongReadLen=>n | determines the maximum size of an individ- ual data field in bytes (0: do not even read long fields)
LongTruncOK=>0/1 | specifies whether data fields that are too long should be truncated (1) or whether an error should be triggered (0)

Terminate Connection

$dbh->disconnect();

terminate connection to the database

Executing SQL Commands, Evaluating SELECT Queries

Execute Queries Without Return of Data Records

$n = $dbh->do("INSERT ..."); | executes an SQL query without returning records; $n contains the number of records that were changed, or 0E0 if no records were changed, or - 1 if the number is unknown, or undef if an error has occurred
$n = $dbh->do($sql, \%attr, @values); | executes a parameterized query; @values con- tains values for the wild card expressed in the SQL command by ?; these values are handled automatically with quote( ); %attr can contain optional attributes (otherwise, specify undef )
$id = $dbh->{' mysql_insertid'}; | returns the AUTO_INCREMENT value of the last record to be inserted (caution: the attribute mysql_insertid is MySQL-specific)

Execute Queries with Return of Data Records

$sth = $dbh->prepare("SELECT . . . "); | prepares an SQL query (generally SELECT queries); all further operations proceed with the help of the statement handle
$sth->execute( ); | executes the query
$sth->execute(@values); | executes a parameterized query; @values con- tains values for the wild card expressed in the SQL command by ?;
$sth->fetchxxx( ); | evaluates the results (see below)
$sth->finish( ); | releases the resources of the statement handle

If a query was executed with prepare and execute and a list of records was returned as result, then this list can be evaluated with a number of fetch methods.

Evaluate List of Data Records

@row = $sth->fetchrow_array( ); | reads the next record into the array @row; if the end of the list is reached or if an error occurs, then @row contains an empty array; access to individual elements proceeds with $row[n] (where for the first column, n=0)
@row = $sth->fetch( ); | equivalent to fetchrow_array( )
$rowptr = $sth->fetchrow_arrayref( ); | equivalent to fetchrow_array( ), but returns pointers to arrays (or undef if the end of the list of records is reached or an error occurs)
$row = $sth->fetchrow_hashref( ); | reads the next record into the associated array $row; if the end of the list of records is reached or if an error occurs, then $row contains the value undef ; access to individual elements proceeds with $row->{' columnname'}, where case sensitivity is in force
$result = $sth->fetchall_arrayref( ); | reads all records and returns a pointer to an array of pointers to the individual records; access to individual elements proceeds with $result->[$row][$col]
$result = $sth->fetchall_arrayref(); | as above, but the records are now associative arrays; access is via $result->[$row]->' columnname'

Bind Variables to Columns (for fetchrow_array)

$sth->bind_col($n, \$var); | binds the column n to the variable $var (where for the first column we have, exceptionally, n=1); the variable is automatically updated when the next record is read; bind_col must be executed after execute; the function returns false if an error occurs
$sth->bind_columns(\$var1, \$var2, . . . ); | equivalent to bind_col, except that variables are assigned to all columns of the query; make sure you have the correct number of variables

Metainformation on SQL Commands

$n = $sth->{' NUM_OF_FIELDS'}; | returns the number of result columns (after SELECT)
$n = $sth->{' NUM_OF_PARAMS'}; | returns the number of parameters in queries with wild cards
$sql = $sth->{' Statement'}; | returns the underlying SQL command

Determine Column Names, Data Types, etc., of SELECT Results

$array_ref = $sth->{' NAME'}; | returns a pointer to an array with the names of all columns; evaluation takes place with @$array_ref[$n], where n ranges from 0 to $sth->{' NUM_OF_FIELDS'}-1
$array_ref = $sth->{' NAME_lc'}; | as above, but names in lowercase
$array_ref = $sth->{' NAME_uc'}; | as above, but names in uppercase
$array_ref = $sth->{' NULLABLE'}; | specifies for each column whether NULL may be stored there (1) or not (0); if this information cannot be determined, then the array contains the value 2 for this column
$array_ref = $sth->{' PRECISION'}; | specifies the precision in the sense of ODBC (here is meant the maximum column width)
$array_ref = $sth->{' SCALE'}; | specifies the number of decimal places for floating-point numbers
$array_ref = $sth->{' TYPE'}; | specifies the data type of all columns in the form of numerical values; the values relate to the ODBC standard; tests determined the following values: CHAR: 12, INT: 4, TEXT/BLOB: - 1, DATE: 9, TIME: 10, TIMESTAMP: 11, FLOAT: 7, DECIMAL: 3, ENUM/SET: 1

Shorthand Notation

@row = $dbh->selectrow_array($sql); | corresponds to a combination of prepare, execute, and fetchrow_array; the result is an array of the first result data record; access to further records is not possible
$result = $dbh->selectrow_array($sql); | as above, but $result contains the value of the first column of the first result record
$result = $dbh->selectall_arrayref($sql); | corresponds to prepare, execute, and fetchall_arrayref ; for evaluation of $result, see fetchall_arrayref

Marking Special Characters in Character Strings and BLOBs with the Backslash

$dbh->quote($data);

prefixes the contents of $data between single quotes, prefixes \ and ' with \, and replaces zero- bytes by \0; if $data is empty (undef ), then quote( ) returns the character string NULL

Error Handling

Methods for Error Handling

$h->err( ); | returns the error number of the last error (0: no error)
$h->errstr( ); | describes the last error (empty character string: no error)
$h->trace($n [, $filename]); | logs all internal MySQL data accesses and redirects output to STDERR or the given file; n specifies the degree of detail to be logged (0 deactivates logging, 9 logs everything)

Auxiliary Functions

DBI Functions

@bool = DBI::looks_like_a_number(@data); | tests for each element in the array @data whether it is a number and returns true or undef in the result array
$result = DBI::neat($data [, $maxlen]); | formats the character string contained in $data in a form suitable for output; character strings are placed in single quotes; non-ASCII charac- ters are replaced by a period; if the character string is longer than $maxlen characters (de- fault 400), then it is truncated and terminated with . . .
$result = DBI::neat_list(\@listref, $maxlen, $sep); | as above, but for an entire array of data; the individual elements are separated by $sep (default ",")

$dbh Methods

$ok = $dbh->ping( );

tests whether the connection to MySQL still exists and returns true or false accordingly

MySQL-Specific Extension of the DBD::mysql Driver

If you use the DBI module for access to MySQL databases, then there are some supplementary functions available via DBI methods and attributes, of which we shall now describe some of the most important. The use of these functions can simplify programming and can make Perl programs more efficient. However, the code will no longer be portable; that is, a later change to another database system will require additional work.

Administrative Functions Based on a Separated Connection

$drh = DBI->install_driver(' mysql'); | returns a driver handle
$drh->func(' createdb', $database, $host, $user, $password, ' admin'); | creates a new database; a new connection is used for this
$drh->func(' dropdb', $database, $host, $user, $password, ' admin'); | deletes a database
$drh->func(' shutdown', $host, $user, $password, ' admin'); | shuts down the MySQL server
$drh->func(' reload', $host, $user, $password, ' admin'); | reinputs all MySQL tables (including the mysql tables with privilege management)

Administrative Functions Within the Current Connection

$dbh->func(' createdb', $database, ' admin'); | creates a new database
$dbh->func(' dropdb', $database, ' admin'); | deletes a database
$dbh->func(' shutdown', ' admin'); | shuts down the MySQL server
$dbh->func(' reload', ' admin'); | reinputs all MySQL tables

$dbh Attributes

$id = $dbh->' mysql_insertid'; | returns the AUTO_INCREMENT value of the most recently inserted data record
$info = $dbh->' info'; | after certain special SQL commands is sup- posed to return a character string with information about the command (e.g., after an UPDATE command: Rows matched: 13 Changed: 13 Warnings: 0); however, several tests of this attribute were unsuccessful
$threadid = $dhb->' thread_id'; | returns the thread ID number of the current connection to MySQL

$sth Methods and Attributes

$sth->rows( ); | returns after SELECT queries the number of data records found by SELECT; caution: this does not work if $sth->' mysql_use_result'=1 holds
$sth->' mysql_store_result'=1; | activates mysql_store_result, so that with SELECT queries all results are stored temporarily on the client computer (default setting)
$sth->' mysql_use_result'=1; | activates mysql_use_result, so that with SELECT queries only a single record is stored temporarily on the client

$sth Attributes for Determining Metadata on SELECT Results

$ar_ref = $sth->' MYSQL_IS_BLOB'; | returns a pointer to an array whose values specify whether the column contains BLOBs; evaluation with @$ar_ref[$n], where n ranges from 0 to $sth->' NUM_OF_FIELDS'-1
$ar_ref = $sth->' MYSQL_IS_KEY '; | specifies whether the columns are indexed
$ar_ref = $sth->' MYSQL_IS_NOT_NULL'; | specifies whether the attribute NOT NULL holds for the columns
$ar_ref = $sth->' MYSQL_IS_NUM'; | specifies whether numerical data are stored in the columns
$ar_ref = $sth->' MYSQL_IS_PRI_KEY '; | specifies which columns are part of the primary index
$ar_ref = $sth->' MYSQL_MAX_LENGTH'; | specifies the maximum column width of the query results
$ar_ref = $sth->' MYSQL_TABLE'; | specifies the underlying table names for all columns
$ar_ref = $sth->' MYSQL_TYPE_NAME'; | specifies the names of the data types for all columns


JDBC (Connector/J)

In order to be able to access MySQL under Java, a JDBC driver for MySQL must be installed. This book assumes that you are using Connector/J version 3.n. If this assumption is satisfied, you can use numerous classes and methods of JDBC (Java Database Connectivity) with the names java.sql.* and javax.sql.*. The following tables assemble only the most important classes and methods of JDBC. There is simply no room for a complete reference to this complex library for database programming.

Establishing a Connection

Connection with DriverManager

import java.sql.*; | enables direct access to the JDBC base classes
Class.forName( | loads and registers Connector/J, the MySQL
"com.mysql.jdbc.Driver"). newInstance( ); | driver for JDBC
Connection conn = DriverManager.getConnection( "jdbc:mysql://hostname/dbname", "username", "password"); | creates a connection to the database dbname on the computer hostname; in the connection string, a large number of additional optional parameters may be passed, the most important of which appear in a table of Chapter 15

Connection with DataSource (since Java 2, Version 1.4)

import java.sql.*; import javax.sql.*; | enables direct access to the JDBC base and extension classes
com.mysql.jdbc.jdbc2.optional. MysqlDataSource ds = new com.mysql.jdbc.jdbc2.optional. MysqlDataSource( ); ds.setServerName("hostname"); ds.setDatabaseName("dbname"); Connection conn = ds.getConnection( "username", "password"); | creates an object of the class com.- mysql.jdbc.jdbc2.optional.MysqlDataSource, sets the host and database names, and finally establishes the connection;
ds.setUrl(" . . . "); | to be used instead of setServerName and setDatabaseName for setting various con- nection parameters; the syntax of the connection string (URL) is the same as for DriverManager.getConnection

Transactions

conn.setAutoCommit(false); | enables transactions (only, of course, if the MySQL tables are transaction-capable)
conn.commit( ); | confirms all SQL commands executed in the last transaction and starts the next transaction
conn.rollback( ); | aborts the commands of the last transaction and begins a new transaction

Executing SQL Commands

SQL Commands (Statement)

Statement stmt = conn.createStatement( ); Statement stmt = conn.createStatement( java.sql.ResultSet. TYPE_FORWARD_ONLY, java.sql.ResultSet. CONCUR_READ_ONLY); stmt.setFetchSize( Integer.MIN_VALUE); | creates a statement object, necessary to execute an SQL command defines a statment object for forward only ResultSets
Statement stmt = conn.createStatement( java.sql.ResultSet. TYPE_SCROLL_SENSITIVE, java.sql.ResultSet. CONCUR_UPDATABLE); | defines a statement object for a variable ResultSet
int n = stmt.executeUpdate( "INSERT . . . "); | executes INSERT, UPDATE, and DELETE com- mands; the return value specifies the number of changed data records
ResultSet res = stmt.executeQuery( "SELECT . . . "); stmt.addBatch("INSERT . . . "); stmt.addBatch("INSERT . . . "); int[ ] n = stm.executeBatch( ); | executes a SELECT query and returns as result a ResultSet object collects a number of SQL commands and exe- cutes them as a group; executeBatch returns an int field that specifies the number of changed records

Determining AUTO_INCREMENT IDs After INSERT Commands

stmt.executeUpdate("INSERT . . . "); | the starting point for the following three variants
ResultSet newid = stmt.getGeneratedKeys( ); if(newid.next( )) { int id = newid.getInt(1); } | getGeneratedKeys returns a ResultSet object with the most recently generated ID num- ber(s); normally, that is, after a usual INSERT command, res contains exactly one ID num- ber that is read with next( ) and getInt(1); getGeneratedKeys is available since Java 2, version 1.4
long id = ((com.mysql.jdbc. Statement)stmt).getLastInsertID( ); | getLastInsertID also returns the ID number; however, the method is Connector/J-specific and not portable
ResultSet newid = stmt.executeQuery( "SELECT LAST_INSERT_ID( )"); if(newid.next( )) { id = newid.getInt(1); } | Here the ID number is returned with a separate SQL command; note that the command is executed in the same transaction

Executing PreparedStatments

PreparedStatement pstmt = conn.prepareStatement( "INSERT . . . (?, ?)"); | declares an SQL command with two parameters indicated by question marks
pstmt.setString(1, "O'Reilly"); pstmt.setInt(2, 7878); | passes the parameter; there are numerous methods in addition to setString and setInt, e.g., setNull, setDate, setTime, setFloat, setBi- naryStream; to these methods are passed the parameter number (beginning with 1) and the actual data
int n = pstmt.executeUpdate( ); ResultSet res = pstmt.executeQuery( ); pstmt.addBatch( ); int n pstm.executeBatch( ); | executes the command(s); the methods have the same meaning as for the Statement class

Changing ResultSets

res.deleteRow( ); | deletes the active record
res.updateXxx(n, data); | first changes the specified columns of the
res.updateRow( ); | active record and then stores the changes
res.moveToInsertRow( ); | inserts a new record, changes its columns, and
res.updateXxx(n, data); | then stores the changes
res.insertRow( ); |

Evaluating SELECT Queries

Evaluating ResultSet

res.getInt(n); res.getString(n); res.getBytes(n); . . . | returns the data field of column n of the currently active record; instead of the col- umn number (1 for the first column), the name of the column may be specified, e.g., getDate("birthdate")
res.wasNull( ); | tests whether the most recently read data field was NULL; this test is necessary with elementary Java data types that cannot store the value NULL and instead contain 0; wasNull offers the only way of distinguishing between 0 and NULL
res.getBinaryStream(n); | returns an InputStream object for bytewise reading of binary data
res.getCharacterStream(n); | returns a Reader object for character-wise reading of binary data
res.getBlob/getClob(n); | returns a Blob or Clob object for reading binary data

ResultSet Navigation

res.next( ); | makes the next record in ResultSet the active record; the method returns false if there are no more records
res.first( ); | activates the first record; the method returns false if the ResultSet contains no records
res.previous( ); | activates the previous record
res.last( ); | activates the last record
res.beforeFirst( ); | places the record cursor before the first record; the ResultSet object is thereby in the same condition as immediately after executeQuery; now the first record can be activated with next
res.afterLast( ); | places the record cursor after the last record; previous activates the last record
res.isFirst( ); res.isLast( ); | tests whether the current record is the first/last
int n = res.getRow( ); res.isLast( ); | returns the number of the active record (1 for the first record)
res.absolute(n); | activates record n

Metadata on ResultSet

ResultSetMetaData meta = | returns a ResultSetMetaData object that gives
res.getMetaData( ); | information about the SELECT result
meta.getColumnCount( ); | returns the number of columns
meta.getColumnName(i); | returns the name (String) of column i
meta.getColumnType(i); | returns the data type of column i; the re- sult is an int with one of the constants from java.sql.Types
meta.getColumnTypeName(i); | returns the name (String) of the data type of column i
meta.IsNullable(i); | tells whether the column may contain NULL
meta.IsAutoIncrement(i); | tells whether it is an AUTO_INCREMENT column

<sql:xxx> Tags for Database Access in JSP Projects

Establishing the Connection

<%@ taglib uri="http://java.sun.com/jstl/sql" prefix="sql" %> | tells that SQL tags are used in the JSP file (for this to work, the relevant parts of the JSTL must be correctly installed in the project directory; see Chapter 15)
<sql:setDataSource var="conn" driver="com.mysql.jdbc.Driver" url="jdbc:mysql://hostname/dbname" user="username" password="password" /> | creates the connection to the database db- name on computer hostname and stores the connection information in the variable conn

Executing SQL Commands

<sql:query var="res" dataSource="${conn}" [startRow ="n" maxRows="m"] > SELECT . . . </sql:query> | for the database connection conn, executes the specified SELECT query and stores the result in the variable res; optionally, the result list can be limited via startRow and maxRows; in the query, parameters are denoted by ?; these parameters must then be specified by <sql:param> or <sql:dataParam>
<sql:update var="count" dataSource="${conn}" > UPDATE/INSERT/DELETE . . . </sql:update> | executes an UPDATE, INSERT, or DELETE command and stores the number of changed records in the variable count; JSTL appears to offer no mechanism to determine the AUTO_INCREMENT number of the last- inserted record (at least I have not found one); you must therefore execute a separate query SELECT LAST_INSERT_ID( ); if you are working with InnoDB, then the INSERT and SELECT commands must be executed within a transaction
<sql:param value="${var}" /> | passes to a parameter the content of the vari- able var; <sql:param> must be placed within <sql:query> or <sql:update>, where the order of the ? signs in the SQL command must be adhered to; Example: <sql:update dataSource="${conn}" > DELETE FROM x WHERE a=? AND b=? <sql:param value="${varA}" /> <sql:param value="${varB}" /> </sql:update>
<sql:dataParam value="${var}" type="date" / "time" / "timestamp" /> | specifies a parameter for the date (type="date", time ("time"), or both ("timestamp")
<sql:transaction> <sql:query . . . > . . . </sql:query> <sql:update . . . > . . . </sql:update> <sql:update . . . > . . . </sql:update> </sql:transaction> | executes several SQL commands within a transaction

Evaluating SELECT Results

<c:forEach var="row" items="${res.rows}" > <c:out value="${row.colname}" /> . . . </c:forEach> | using <c:forEach>, the variable row runs through all rows of the result res; row.columnname gives access to a single column of the current record
<c:forEach var="row" items="${res.rowsByIndex}" > <c:out value="${row[0]}" /> . . . </c:forEach> | as above, but access to the columns is now via index numbers (0 for the first column)
${res.rowCount} | returns the number of found data records


C API

The following tables assemble the most important functions and structures of the C API.

Data Structures

Basic Structures

MYSQL *conn; | structure with connection data
MYSQL_RES *result; | structure with the results of a SELECT query
MYSQL_ROW row; | pointer to the results of a row (i.e., of a data record)
MYSQL_ROW_OFFSET roffset; | pointer to a record within the result list
MYSQL_FIELD *field; | structure for describing a column (column name, data type, number of digits, etc.); details in the next table
MYSQL_FIELD_OFFSET foffset; | offset within a record (0 for the first column, 1 for the second, etc.)
my_ulonglong n; | 64-bit integer; some of the MySQL functions described below return results of this data type

In all further syntax tables, the variables conn, result, row, field, roffset, and foffset will be used as if declared in the above table. Note that MYSQL_ROW is already a pointer and is therefore declared without * .

Elements of the MYSQL_FIELD Structure

char *name; | name of the column
char *table; | name of the table from which the column comes; if the column was computed or an ALIAS was used, then table points to a character string with the formula or the ALIAS name
char *def; | default value of the column or NULL
enum enum_field_types type; | data type of the column; these are the choices: FIELD_TYPE_BLOB, FIELD_TYPE_CHAR, FIELD_TYPE_DATE, FIELD_TYPE_DATETIME, FIELD_TYPE_DECIMAL, FIELD_TYPE_DOUBLE, FIELD_TYPE_ENUM, FIELD_TYPE_FLOAT, FIELD_TYPE_INT24, FIELD_TYPE_LONG, FIELD_TYPE_LONGLONG, FIELD_TYPE_NULL, FIELD_TYPE_SET, FIELD_TYPE_SHORT, FIELD_TYPE_STRING, FIELD_TYPE_TIME, FIELD_TYPE_TIMESTAMP, FIELD_TYPE_TINY, FIELD_TYPE_YEAR
unsigned int length; | length of the column according to the column definition
unsigned int max_length; | maximal length of a column within the query result; the value is always 0 if you use mysql_use_result( )
unsigned int flags; | additional information for describing the column: AUTO_INCREMENT_FLAG, BINARY_FLAG, MULTIPLE_KEY_FLAG, NOT_NULL_FLAG, PRI_KEY_FLAG, UNIQUE_KEY_FLAG, UNSIGNED_FLAG, ZEROFILL_FLAG
unsigned int decimals; | number of places after the decimal point in DECIMAL columns (e.g., 5 for DECIMAL(10,5))

Connection and Administration

Establishing a Connection

MYSQL *conn; conn = mysql_init(NULL);

initializes the MYSQL data structure

mysql_options(conn, option, "value");

sets additonal options for the connection; an option with one of the following values must be passed: MYSQL_OPT_CONNECT_TIMEOUT, MYSQL_OPT_LOCAL_INFILE, MYSQL_OPT_NAMED_PIPE, MYSQL_INIT_COMMAND, MYSQL_READ_DEFAULT_FILE, MYSQL_READ_DEFAULT_GROUP;

with some of these options, "value" can be used to specify the desired value; for setting several options, the function must be called repeatedly; mysql_options must be exectued before mysql_real_connect

mysql_real_connect(conn, "hostname",

makes a connection to the database and re-

"username", "password", "dbname",

turns NULL in case of error; flags can contain a

portnum, "socketname", flags);

combination of the following values:

CLIENT_COMPRESS,

CLIENT_FOUND_ROWS,

CLIENT_IGNORE_SPACE,

CLIENT_INTERACTIVE,

CLIENT_NO_SCHEMA,

CLIENT_ODBC,

CLIENT_SSL

mysql_change_user(conn, "username", "password", "dbname");

changes the user and the default database for an existing connection

mysql_change_db(conn, "dbname");

changes the default database; the function assumes that the user has access to the database

mysql_ping(conn);

tests whether the connection still exists; if not, the connection is recreated; returns 0 as result if an active connection exists

mysql_close(conn);

closes the connection

Acquiring Information on the Current Connection

mysql_characterset_name(conn); | returns a character string with the default character set of the connection
mysql_get_client_info( ); | returns a character string with information on the version of the client library in use (e.g., "4.0.12")
mysql_get_server_info(conn); | returns a string with information on the version of the server (e.g., "4.0.12")
mysql_get_host_info(conn); | returns a string with information on the con- nection to the server (e.g., "localhost via UNIX socket")
mysql_get_proto_info(conn); | returns the version number (unsigned int) of the connection protocol, e.g., 10 or 11
mysql_info(conn); | returns a string with information on the ex- ecution of the last INSERT, UPDATE, LOAD DATA, or ALTER TABLE command (e.g., "Rows matched: 3 Changed: 3 Warnings: 0")
mysql_stat(conn); | returns a string with the server status (number of threads, number of open tables, etc.)
mysql_thread_id(conn); | returns the number (unsigned long) of the thread that the current connection is processing on the server

Adminstrative Functions

mysql_kill(conn, n); | ends the thread specified by n (requires the Process privilege)
mysql_shutdown(conn); | shuts down the server (execution requires the Shutdown privilege)

Error-Handling

mysql_errno(conn); | returns the error number (unsigned int) for the most recently executed command (or 0 if there was no error)
mysql_error(conn); | returns a string with the error message (or an empty string "" if there was no error)

Executing and Evaluating SQL Commands

Execution of SQL Commands

mysql_query(conn, "SELECT . . . "); | executes the specified command and returns 0 if the server accepts the command without triggering an error
mysql_real_query(conn, "SELECT . . . ", len); | like mysql_query, except that the SQL command may now contain the zero-byte (e.g., to store BLOBs); for this, the length of the string must be specified explicitly
mysql_affected_rows(conn); | returns the number (data type my_ulonglong) of changed records after a DELETE, INSERT, or UPDATE command; the function does not return the number of results of a SELECT command
mysql_insert_id(conn); | returns the AUTO_INCREMENT value (data type my_ulonglong) of the last record created via INSERT

Processing SELECT Results

result = mysql_store_result(conn); | transfers all results from the server to the client and stores then in the MYSQL_RES structure
result = mysql_use_result(conn); | represents an alternative to mysql_store_result; the transfer of individual records is only prepared; no data are actually transferred
mysql_num_fields(result); | returns the number of columns of the result
row = mysql_fetch_row(result); | transfers the next record to a MYSQL_ROW structure; the function returns NULL if there are no further records available, i.e., if all records have already been processed
row[n]; | returns a 0-terminated string with the content of column n of the current record; note that row[n] can contain NULL; if you are processing binary data with zero-bytes, you must make the call with mysql_fetch_lengths to determine the size of the data field
mysql_free_result(result); | releases the result structure; if you are working with mysql_use_result, the result is only now released by the server

Processing SELECT Results: Metainformation on Columns and Fields

mysql_fetch_lengths(result); | returns for the current record an unsigned long field with the length of the result string in row[n]
field = mysql_fetch_field(result); | returns a description (MYSQL_FIELD struc- ture) of the data type of a column; the first call returns the data for the first column, the next for the second column, etc.; the function returns NULL after all columns have been run through; the elements of the MYSQL_FIELD structure were described previously
field = mysql_fetch_field_direct(result, n); | returns a description of column n (0 for the first column, etc.)
mysql_fetch_fields(result); | returns a description of all columns as a MYSQL_FIELD field

The following functions can be used only if you are using mysql_store_result (and are not working with mysql_use_result).

Processing SELECT Results: Additional Functions for mysql_store_result

mysql_num_rows(result); | returns the number of found records
mysql_data_seek(result, n); | moves the row cursor to record n (0 for the first record); then mysql_fetch_row must be executed to reinput data
roffset = mysql_row_tell(result); | returns a pointer (a sort of bookmark) to the current record
mysql_row_seek(result, roffset); | moves the row cursor to a particular location within the result list; then mysql_fetch_row must be executed to reinput data; roffset must be determined earlier with mysql_row_tell

Auxiliary Functions

n = mysql_real_escape_string( conn, dest, src, srclen); | copies the string src into the string dest while replacing the special characters with \ char- acter combinations (\0, \b, \t, \", \', etc.); dest is terminated with a zero-byte; srclen specifies the number of characters in src; dest must be previously initialized with a string of the appropriate length; to be able to copy a string full of special characters without error, dest must offer place for srclen*2+1 characters; the function returns the number of characters in dest (without the zero-byte). If "O'Reilly" is copied, then afterward, src contains the string O\'Reilly, and n the value 9
destpt = strmov(dest, src); | like strcpy, copies the string src to the string dest; the difference is in the return value, which points to the end of the string in dest; this makes it easy to construct a string from several pieces; to be able to use strmov, the files my_global.h and m_string.h must be included before mysql.h


Part V: Appendices

Chapter List

Appendix A: Glossary
Appendix B: MySQL 4.1
Appendix C: Example Files
Appendix D: Bibliography


Appendix A: Glossary

Glossary

This appendix contains a brief description of the most important notions from the world of databases and related areas.
Character Set
A character set is a specification that tells which characters are represented by which code. Among the oldest character sets is the ASCII (American Standard Code for Information Interchange) character set, in which the letter A is coded by the integer 65.
Client
A client is a program that accesses a central service. In connection with databases, a client denotes a program that accesses a central database. The database is managed by another program, the database server.
Client/Server Architecture
Most modern database systems are based on a server that manages the data and executes SQL commands, and an arbitrary number of clients that access the server over a network. This concept is called client/server architecture. Alternatives to this model are file-server databases such as dBase and Access, in which all clients access the database file directly through a common file system. (This is quite inefficient when many users are trying to edit data simultaneously.)
Database
Generally speaking, a database is an ordered collection of data. In connection with MySQL, a database consists of a number of tables. In common usage, a database can also refer to an entire database system, that is, the program that manages the data. In MySQL, this program is called the database server (MySQL server).
Database Server
A database server is a program that permits access to databases over a network. The server is responsible for the management of the data and for the execution of SQL commands.
Data Record
A data record, or simply record, is a row of a table.
Domain Name
The domain name denotes an entire network. A local network might consist of computers with the names mars.sol, uranus.sol, and jupiter.sol, in which case mars, uranus, and jupiter are computer names (hostnames), and sol is the domain name.
Domain Name Server (DNS)
Internally, communication within a network takes place on the basis of IP numbers (e.g., 192.168.0.27). Since mere mortals do not keep track of such numbers very well, each computer also has a computer name, composed of host name and domain name (e.g., uranus.sol).
Whenever such a name is used, the running program must determine the associated IP number. Sometimes, the converse case arises, and for a given IP number, the computer name is required. In both situations, the domain name server provides the answer. (Small networks can get by without a DNS. In that case, the association between IP numbers and computer names is determined from static tables, under Unix/Linux in the file /etc/hosts.)
Foreign Key
A foreign key is a unique value (usually an integer) that refers to a particular column of another table. A foreign key thus creates a link (relation) between two tables.
Full-Text Index
A full-text index enables the search for several words in arbitrary order within a text. To make full-text search efficient, a special index is required, known as a full-text index.
Global Assembly Cache (GAC)
This notion is specific to Microsoft. It describes a central directory of all *.dll files for .NET programs. In contrast to the Windows system directory, in the GAC, several different versions of a library can be installed simultaneously without leading to a conflict.
Host Name
The host name is the name of a computer valid over a network. If the complete name of the computer is jupiter.sol, then jupiter is the host name and sol the domain name.
Hot Backup
What makes a hot backup hot is that during the backup process, the database server can continue to run almost completely undisturbed. (During a normal backup, on the other hand, no changes in the data may be made.) Hot backups can currently be made in MySQL only with a commercial auxiliary program that works only for InnoDB tables.
Include File
An include file is a part of program code. It is read by the compiler (e.g., C) or during code execution (e.g., PHP).
Index
In database-speak, an index, or its synonym key,isanordered list of the content of a table. An index serves primarily to speed up access to individual records: Instead of searching through all records, it suffices to search the index. Then, the desired record can be read directly. The use of an index is comparable to that of an index in a book.
InnoDB Tables
In MySQL, tables can be stored on the hard drive in more than one format, including MyISAM and InnoDB. InnoDB tables possess some additional functionality over MyISAM tables, such as transactions and foreign key constraints. The name InnoDB is derived from the name of the company Innobase, with DB being short for database.
Internet Service Provider (ISP)
An ISP is a company that provides technical support and hosting of web sites. ISPs are used by commercial enterprises and private individuals who wish to maintain web sites on the Internet but do not themselves possess a permanent or sufficiently high-speed Internet connection. Be careful in your choice of ISP that it supports the programs presented in this book (Apache, PHP, MySQL, etc.). There are, of course, ISPs that offer Microsoft products instead.
Key
A key is an ordered index of a table. See also the entry of the synonym Index.
Logging
Logging refers to the recording of SQL commands in a file. The motivation for logging is usually to have available all the changes to the database made since the last backup so that the database can be recreated in the event of disaster. With MySQL, logging is also a prerequisite for replication.
MyISAM Tables
The MySQL server can store tables in various formats. The most important such format is called MyISAM, which stands for indexed sequential access method, referring to the method of access to data records within a file with the aid of an index. An alternative to MyISAM tables is InnoDB tables.
Named Pipes
Named pipes represent a mechanism whereby two programs can communicate under Windows NT/2000/XP. Data exchange takes place according to the principle of first in, first out (FIFO).
Normal Form
A database generally consists of a number of tables. A goal in the design of tables is to avoid redundancy. This goal is supported by particular rules. When the first n such rules are adhered to, the database is said to be in nth normal form.
Normalization
This term refers to the optimization of the database design to avoid redundancy. The goal is generally to achieve third normal form.
PHP
PHP is a script language for programming dynamic web sites. The program code is located in *.php files. These files are executed by the PHP interpreter at each access over the web server. The resulting HTML document is passed on to the web server.
Port
Network packets that are sent using the TCP/IP protocol are always addressed to a particular port. The port number is thus part of the address to which the packet is sent. Ports help in categorizing network traffic. For example, for data exchange between web browsers and web servers, port 80 is used. MySQL uses port 3306 by default.
Primary Index (Primary Key)
The primary index identifies each record of a table uniquely. Normally, the primary key field is an integer column. Whenever a new record is inserted, the database server determines a new, unique, value for this column.
Privilege
In MySQL nomenclature, a privilege is a right to execute particular database operations (e.g., read or change data in a particular table).
Query
In database language, a query is generally a SELECT command. The syntax of this command is part of the language SQL (structured query language). SQL specifies how commands to the database server must be structured.
Query Cache
In many database applications, it happens that the same queries must be executed over and over. To speed up such queries, MySQL possesses, since version 4.0, a query cache, that is, temporary storage in which queries and their results can be stored. Queries thus stored can be answered much more rapidly than otherwise. However, this temporary storage must be deleted at each change in the underlying table.
Referential Integrity
Referential integrity holds if between linked tables, no reference points to nowhere. (For example, if referential integrity is satisfied for a library database, then there cannot exist a book title whose associated publisher does not exist in the database. On the other hand, if referential integrity has been damaged, then there might well be a publisher number in the titles table that does not exist in the publisher table.)
Many database systems can automatically check referential integrity when changes to a database are made. This requires the setting up of foreign key constraints, otherwise known as integrity rules. MySQL currently supports foreign key constraints only for InnoDB tables.
Relations
The most characteristic feature of relational databases is that a number of tables in a database can be linked based on their contents. These links are also called relations. The links are referenced in SQL commands by the JOIN operator or with WHERE conditions.
Replication
Through the process of replication, all changes to a database are carried out not on one, but on several database servers. Thus the same database exists on several servers. The advantages are increased security (if one server goes down) and greater speed (all servers can answer SQL queries, even if only one can make changes).
Schema
The schema (or database schema) describes the layout of a database, that is, the tables, their columns and data types, indexes, links between tables, etc.
Server
A server is a central program that provides services or data to other programs (called clients). Examples of servers are the MySQL server (a database server) and Apache (a web server).
Socket File
A socket file enables communication under Unix/Linux between two programs. A socket file is not a real file, in that it contains no data and has length zero.
Local connections between a MySQL client and the MySQL server generally take place under Unix/Linux over a socket file, since this type of communication is more efficient than the network protocol TCP/IP. (An exception is Java clients, which do not support socket files.) The usual file name is /var/lib/mysql/mysql.sock. This file name can be set via the configuration file /etc/my.cnf.
Stored Procedures
A stored procedure is program code that is executed directly by the database server. Often, stored procedures are formulated in a programming language that extends SQL through the use of control structures (loops, branching, etc.). MySQL currently does not support stored procedures.
Tables
A table is a part of a database. The properties of a table are determined by its columns or fields (name and data type) and indexes. The rows of a table are also called data records. The MySQL server can store tables in several formats (e.g., MyISAM, InnoDB).
Timestamps
The data type TIMESTAMP is characterized by the fact that columns of this type store the current date and time each time a change is made.
Transaction
A transaction is a group of logically related SQL commands. Transaction- capable database systems are able to confirm all the commands of a transaction together or to abort them as a group. That is, a transaction is never only partially completed. Transactions thereby contribute to data security.
Transactions can also improve efficiency in that there is no longer the need to lock entire tables with LOCK during the execution of several SQL commands.
MySQL currently supports transactions only for particular table formats, such as InnoDB.
Trigger
This term refers to SQL code (stored procedures) that is automatically executed as a result of particular actions (e.g., upon an UPDATE or INSERT command). Triggers can be used, for example, to ensure that certain rules are followed in changing data. MySQL currently does not support triggers.
Unicode
Unicode is a character set in which every character is represented by a 16-bit integer. Thus almost all the special characters of the world's languages can be represented in this character set. MySQL supports the storage of Unicode character sets since version 4.1.
Within Unicode, there are various formats that specify the byte order of the 16-bit integer. Common formats are UTF8 (1 to 3 bytes, depending on the character code, avoids code 0) and UTF16 (2 bytes for each character code).
Uniform Resource Locator (URL)
This unwieldy term denotes Internet addresses (e.g., http://www.kofler.cc/mysql/ or ftp://ftp.mysql.com). Additional information can be provided in a URL, such as in http://my.company/page.html?var=123 or ftp://username.company/directory.
Views
A view is an SQL query that enables a special view of several tables of a database. Views are used, for example, to enable simple access to particular parts of a table (e.g., for reasons of security if only part of a table is to be processed). Currently, MySQL does not support views.
Web Server
A web server is a program that makes HTML pages available over the Internet. The most popular web server today (current market share about sixty percent) is Apache.


Appendix B: MySQL 4.1

Overview

MySQL 4.1 was officially introduced in January 2003. As I was completing this appendix in July 2003, an alpha version of MySQL 4.1.0 had been available for several months on the MySQL web site. In order to be able to present the most current information possible, I downloaded the developer's version of the source code (as of 28 July 2003) and compiled it under Red Hat Linux 9. The information in this appendix is thus based on a version that lies somewhere between 4.1.0 and 4.1.1.

The first stable MySQL 4.1.n will probably not appear until 2004. By then, there are likely to be considerable changes from what appears here.


SubSELECTs

The improvement desired most by many MySQL users relates to the SELECT command. MySQL finally supports the syntax for formulating nested SELECT commands. There are three syntactic variants:

  • SELECT ... WHERE col = (SELECT ... )

    In this variant, the second SELECT query must return a single value (one row, one column). This value is used for the comparison col = ... . (Other comparison operators are permitted, such as col> ... , col <= ... , col <> ... .)

  • SELECT ... WHERE col IN (SELECT ... )

    In this variant, the second SELECT query may return a list of individual values. This list is then processed in the form SELECT ... WHERE col IN (n1, n2, n3). Instead of IN, one may also use NOT IN.

  • SELECT ... WHERE EXISTS (SELECT ... )

    In this variant, the second query is executed for each found record from the first SELECT query. A record from the first SELECT query remains in the result list only if the second query returns a result (at least one record). In this variant as well, the negation operator NOT may be used.

    EXISTS constructions are generally useful only when the two SELECT commands are linked by a WHERE condition (as with a JOIN operation). We will give an example shortly.

SELECT commands can also be used in the WHERE condition of UPDATE and DELETE commands to determine what records have been altered or deleted.

Examples: The following examples are based on the database mylibrary. The first command determines the number of titles published by Addison-Wesley:

USE mylibrary
SELECT COUNT(*) FROM titles WHERE publID =
  (SELECT publID FROM publishers WHERE publName = "Addison-Wesley")

The next command searches for titles for which no author has been stored:

SELECT title FROM titles
WHERE titleID NOT IN (SELECT titleID FROM rel_title_author)
SELECT title FROM titles WHERE NOT EXISTS
 (SELECT * FROM rel_title_author
  WHERE titles.titleID = rel_title_author.titleID)

The following command deletes all entries from the table rel_title_author that do not refer to existing records in authors or titles:

DELETE FROM rel_title_author
WHERE authID NOT IN (SELECT authID FROM authors) OR
     titleID NOT IN (SELECT titleID FROM titles)
Pointer

Additional subSELECT examples appear in Chapter 7. There, it is shown how equivalent queries can be composed in MySQL 4.0. (SubSELECTs simplify the construction of many commands. In most cases, however, it is possible to create equivalent commands using JOIN operations. This holds as well for the commands just presented.)

Limitations: The version of MySQL that I tested was incapable of evaluating LIMIT in subSELECT commands (e.g., SELECT ... WHERE ... IN (SELECT ... LIMIT 10)).


Working with Character Strings

The management of character sets has fundamentally changed in MySQL 4.1. The following list summarizes the most important changes:

  • While MySQL 4.0 uses a single character set for the entire server (that is, for all databases), now a separate character set can be used for each database, indeed, for each table and each column within a table.

  • The character set and sort order (collation) have been separated. With many character sets, one may now choose between two or more sort orders.

    The character set is responsible only for the encoding of characters, for changing between uppercase and lowercase (UCASE, LCASE), and transformation into Unicode. Furthermore, the character set determines the type of a character (e.g., number, letter, or white space).

    The sort order influences comparison operations like =, <, and >, which also appear in ORDER BY, GROUP BY, DISTINCT, MIN, MAX, etc.

  • MySQL now supports Unicode, and in fact, in two formats: UTF8 and UCS2. In both cases, a 16-bit code is assigned to each character. The formats vary in how these codes are stored.

    With UCS2, 2 bytes are always used. The higher-valued byte is stored first. The character A, with code 65, is thus stored with 2 bytes with values 0 and 65. Thus each character requires exactly 2 bytes. The result is that with ASCII text, every second byte is a zero-byte. (UCS2 is also a synonym for UTF16; both denote the same Unicode format.)

    With UTF8, the number of bytes used for a character depends on the code. ASCII characters (code less than 128) are stored with a single byte, while other characters use 2 to 3 bytes. UTF8 has two advantages: First, problems with the zero-byte are avoided (important for programming in C, which assumes that character strings end with the zero-byte). Second, the storage of English text is much more compact.

The default character set in MySQL remains latin1 with sort order latin1_swedish_ci. If the new string functions are not used, then nothing should change in MySQL's sorting behavior.

Warning

For text columns declared with VARCHAR(n) or CHAR(n), currently n specifies the number of bytes. However, depending on the character set, you may only be able to store many fewer than n characters. With UCS2 the number is simply n/2, while with UTF8, the number varies, since different characters are coded with different numbers of bytes. It is possible that the column length in future versions of MySQL will be oriented to the character set, so that with CHAR(10) CHARSET utf8, a column with thirty bytes is reserved, so that in the worst case, ten UTF8 characters will have sufficient space.

String functions like LEFT, MID, and their kin function without problems with Unicode. MID(column, 3, 1) thus returns the third character (not the third byte). However, caution is called for in determining the length of a character string: LENGTH(column) returns the number of bytes, while CHAR_LENGTH(col) returns the number of characters.

Determining the available character sets and sort orders: The new command SHOW CHARACTER SET returns a list of all character sets and their default sort orders. The column Maxlen gives the maximum number of bytes required to store a character. The following table gives the sorted result of the command SHOW CHARACTER SET:

SHOW CHARACTER SET

Charset | Description | Default collation | Maxlen
armscii8 | ARMSCII-8 Armenian | armscii8_general_ci | 1
ascii | US ASCII | ascii_general_ci | 1
big5 | Big5 Traditional Chinese | big5_chinese_ci | 2
binary | Binary pseudo charset | binary | 1
cp1250 | Windows Central European | cp1250_general_ci | 1
cp1251 | Windows Cyrillic | cp1251_general_ci | 1
cp1256 | Windows Arabic | cp1256_general_ci | 1
cp1257 | Windows Baltic | cp1257_general_ci | 1
cp850 | DOS West European | cp850_general_ci | 1
cp852 | DOS Central European | cp852_general_ci | 1
cp866 | DOS Russian | cp866_general_ci | 1
dec8 | DEC West European | dec8_swedish_ci | 1
euckr | EUC-KR Korean | euckr_korean_ci | 2
gb2312 | GB2312 Simplified Chinese | gb2312_chinese_ci | 2
gbk | GBK Simplified Chinese | gbk_chinese_ci | 2
greek | ISO 8859-7 Greek | greek_general_ci | 1
hebrew | ISO 8859-8 Hebrew | hebrew_general_ci | 1
hp8 | HP West European | hp8_english_ci | 1
keybcs2 | DOS Kamenicky Czech-Slovak | keybcs2_general_ci | 1
koi8r | KOI8-R Relcom Russian | koi8r_general_ci | 1
koi8u | KOI8-U Ukrainian | koi8u_general_ci | 1
latin1 | ISO 8859-1 West European | latin1_swedish_ci | 1
latin2 | ISO 8859-2 Central European | latin2_general_ci | 1
latin5 | ISO 8859-9 Turkish | latin5_turkish_ci | 1
latin7 | ISO 8859-13 Baltic | latin7_general_ci | 1
macce | Mac Central European | macce_general_ci | 1
macroman | Mac West European | macroman_general_ci | 1
sjis | Shift-JIS Japanese | sjis_japanese_ci | 2
swe7 | 7bit Swedish | swe7_swedish_ci | 1
tis620 | TIS620 Thai | tis620_thai_ci | 1
ucs2 | UCS-2 Unicode | ucs2_general_ci | 2
ujis | EUC-JP Japanese | ujis_japanese_ci | 3
utf8 | UTF-8 Unicode | utf8_general_ci | 3

The command SHOW COLLATION LIKE 'name%' returns a list of all sort orders that begin with name. The following table shows the sort orders available for the latin1 character set:

SHOW COLLATION LIKE 'latin1%'

latin1_bin | latin1 | 47 |  | Yes | 0
latin1_danish_ci | latin1 | 15 |  |  | 0
latin1_general_ci | latin1 | 48 |  |  | 0
latin1_general_cs | latin1 | 49 |  |  | 0
latin1_german1_ci | latin1 | 5 |  |  | 0
latin1_german2_ci | latin1 | 31 |  | Yes | 2
latin1_swedish_ci | latin1 | 8 | Yes | Yes | 0

From the table, one can see what the naming convention is: The name of the sort order begins with the name of the character set. The abbreviations _bin, _ci, and _cs at the end of the name stand for binary, case insensitive, and case sensitive. With binary sorting, only the codes of characters are compared. The other variants take conventions of the language in question into account. The ci variant considers upper- and lowercase letters to be equivalent.

The sorting behavior of both german variants is defined by the German Industrial Standard (Deutsche Industrienorm, or DIN). Thus german1 considers ä, ö, ü, and ß as equivalent to a, o, u, and s, while german2 considers ä, ö, ü, and ß as equivalent to ae, oe, ue, and ss.

SQL commands and syntax: To control the character-set functions, there are many new SQL commands and syntactic extensions:

  • The commands CREATE/ALTER DATABASE and CREATE/ALTER TABLE have been extended so that the character set and sort order can be specified for each database, table, and individual text columns (see the command reference in Chapter 18). Here is the syntax for specifying the character set:

    CHARACTER SET csname [COLLATE colname]

    In addition, there are several syntactic variants, such as CHARSET as an abbreviation for CHARACTER SET and the form CHARACTER SET = csname (in the form of an assignment). The following command creates a new table in which Unicode character strings can be stored in all text columns:

    CREATE TABLE test (id INT NOT NULL AUTO_INCREMENT,
                    data1 VARCHAR(30),
                   ...,
                    PRIMARY KEY(id))
CHARSET utf8

  • With CONVERT(s USING charset), the string s is transformed into the desired character set. SELECT HEX(CONVERT('ABCäöü' USING utf8)) returns, for example, the hexadecimal code 414243C3A4C3B6C3BC. For the character set ucs2, the result is 00410042004300E400F600FC.

  • In the specification of character strings, the name of the character set can be prefixed with _, e.g, _latin1 "abcäöü" or _utf8 "abc".

    If the sort order is to be specified in comparisons, this can be given with COLLATE. For example, the following query without COLLATE returns 0 (FALSE), while with COLLATE the result is 1 (TRUE). The same holds as well for comparisons with WHERE and with HAVING:

    SELECT _latin1 "a" COLLATE latin1_german1_ci =
        _latin1 "ä" COLLATE german1_latin1_ci

    COLLATE leads to an error if the desired sort order is unavailable for the given character set.

  • COLLATE can also be used with a variety of other SQL commands:

    SELECT column COLLATE colname [AS aliasname]
ORDER BY column COLLATE colname
GROUP BY column COLLATE colname
DISTINCT column COLLATE colname

  • COLLATE can also be used in aggregation functions (e.g., SELECT MAX(column COLLATE colname) ... ).

  • The new data type NCHAR (for national character) corresponds to CHAR CHARSET utf8. (If you wish to define a UTF8 column, then with NCHAR you avoid having to specify the character set.)

    UTF8 columns of variable length can be declared with NCHAR VARCHAR(n). (The key word NVARCAHR does not exist.)

Pointer

There is much more information on character set management in a new chapter of the MySQL documentation:

http://www.mysql.com/doc/en/Charset.html


A Unicode Example

The following commands show how one can begin experimenting with the new functions. We create a table with two columns, one in the character set latin1, the other in the character set utf8:

USE test
CREATE TABLE t1
  (col1 VARCHAR(50) CHARACTER SET latin1,
  col2 VARCHAR(50) CHARACTER SET utf8)
INSERT INTO t1 (col1) VALUES
  ('abc'), ('Abc'), ('ABC'), ('bar'), ('Barenboim'),
  ('Bär'), ('Bären'), ('bärtig'), ('Ärger')
UPDATE t1 SET col2 = CONVERT(col1 USING utf8)
SELECT *, LENGTH(col2), CHAR_LENGTH(col2) FROM t1
col1 | col2 | LENGTH(col2) | CHAR_LENGTH(col2)
abc | abc | 3 | 3
Abc | Abc | 3 | 3
ABC | ABC | 3 | 3
bar | bar | 3 | 3
Barenboim | Barenboim | 9 | 9
Bär | Bär | 4 | 3
Bären | Bären | 6 | 5
bärtig | bärtig | 7 | 6
Ärger | Ärger | 6 | 5

By default, column col1 uses Swedish sort order:

SELECT col1 FROM t1 ORDER BY col1

col1

abc

Abc

ABC

bar

Barenboim

Bär

Bären

bärtig

Ärger

However, one could just as well use the german1 sort order:

SELECT col1 FROM t1 ORDER BY col1 COLLATE latin1_german1_ci

col1

abc

Abc

ABC

Ärger

bar

Bär

Bären

Barenboim

bärtig

For col2, there are currently no available country-specific sort orders.


Managing Geometric and Geographic Data (OpenGIS)

Traditional tables and indexes are not very suitable for storing geometric and geographic data. For MySQL to be usable with geographic information systems (GIS for short), MySQL 4.1 has been extended with various new data types and a special R-tree index. The implementation adheres largely to the OpenGIS standard:

http://www.mysql.com/doc/en/Spatial_extensions_in_MySQL.html
http://www.opengis.org/

There is insufficient space here for a complete description of the possibilities opened up by these new developments. The following overview will have to serve as a brief introduction.

SQL commands and syntax: The commands CREATE and ALTER TABLE have been extended so that columns of data type GEOMETRY and spatial indexes (SPATIAL INDEX) can be defined:

CREATE TABLE t (id INT NOT NULL AUTO_INCREMENT,
                g GEOMETRY NOT NULL,
                PRIMARY KEY (id),
                SPATIAL INDEX (g))

The spatial index is based internally on R-trees. The basic idea is that for all geometric objects, a minimal bounding box is determined. Then, for the index, groups of neighboring objects are collected within a large rectangle. The result is a treelike index whose root node encompasses all the objects. The smaller rectangles contained within the larger one encompass smaller subsets of the geometric objects. A geometric search for objects can be speeded up considerably by such a structure, because one must search only those rectangles that overlap the search area.

Instead of GEOMETRY, the terms POINT, MULTIPOINT, LINESTRING, MULTI- LINESTRING, POLYGON, MULTIPOLYGON, and GEOMETRYCOLLECTION can be used. In each case, the column has type GEOMETRY. (In MySQL, in contrast to the OpenGIS specification, there do not exist different geometric data types, but only the all-inclusive type GEOMETRY, in which all possible geometric objects can be stored.)

To insert data into a GEOMETRY column, one can use the new functions GeomFromText and GeomFromWBK. The function GeomFromText expects as argument a character string, while GeomFromWBK expects a binary object that conforms to the WBK specification of OpenGIS. (WBK stands for well-known binary representations; it is a format for the binary representation of geometric objects.)

INSERT INTO t (g) VALUES(
  GeomFromText('POINT(10 10)'))
INSERT INTO t (g) VALUES(
  GeomFromText('LINESTRING(0 0, 10 10, 0 20)'))
INSERT INTO t (g) VALUES(
  GeomFromText('POLYGON((30 30, 30 40, 40 40, 40 30, 30 30))'))

To indicate geometric objects with SELECT, you must use the function AsText. If you require binary representation, then instead, use the function AsBinary:

SELECT id, AsText(g) FROM t
id | AsText(g)
1 | POINT(10 10)
3 | LINESTRING(0 0,10 10,0 20)
4 | POLYGON((30 30,30 40,40 40,40 30,30 30))

If you wish to select all objects within a particular geometric region, then you can formulate your WHERE condition using the new function WITHIN:

SELECT id, AsText(g) FROM t
WHERE Within(g, textbfGeomFromText('POLYGON((5 5, 5 15, 15 15, 15 5, 5 5))'))
id | AsText(g)
1 | POINT(10 10)

There are many additional functions for processing geometric objects, and here we can mention only a few representative examples to give you a taste of what is available:

  • GeometryType(geom) determines the type of a GEOMETRY object (e.g., POINT or LINESTRING).

  • X(point) and Y(point) determine the X and Y coordinates of a point.

  • Envelope(geom) determines the bounding box, that is, the minimal circumscribing rectangle for the object. The result is a rectangular POLYGON object.

  • Area(polyg) calculates the area of a polygon.

  • Intersects(g1, g2) tests whether two geometric objects overlap.

Here is an example:

SELECT id, GeometryType(g), AsText(Envelope(g)) FROM t
id | GeometryType(g) | AsText(Envelope(g))
1 | POINT | POLYGON((10 10,10 10,10 10,10 10,10 10))
3 | LINESTRING | POLYGON((0 0,10 0,10 20,0 20,0 0))
4 | POLYGON | POLYGON((30 30,40 30,40 40,30 40,30 30))

Limitations: There are some restrictions to be aware of in the use of OpenGIS functions:

  • GIS data can be stored only in MyISAM tables.

  • Only two-dimensional objects can be managed.

  • Various functions were not yet implemented in the alpha version tested. It is to be expected that many tests will have to be made before one can consider the OpenGIS functions in MySQL to be mature and stable.


Precompiled SQL Commands (Prepared Statements)

The MySQL server and its client library are now capable of first sending an SQL command and then executing it repeatedly with various parameters. At the moment, it is only C programmers who can profit from this capability, but one may expect that the APIs for PHP, Perl, etc., will soon offer equivalent commands.

This type of command execution is useful whenever the identical command (generally INSERT or UPDATE) is to be executed repeatedly with differing parameters. Precompiled commands should be more efficient in such a case, since there would be less data traffic between client and server and since the server can optimize the execution of the command.

It remains, however, to see how noticeable this speed improvement will be in practice. I have not made any speed tests, but I suspect that the difference will be small. I see the principal advantage in the fact that in many cases, more readable program code will be achievable. In particular, as a programmer, you need no longer consider whether special characters appear in character strings or how numbers in strings are transformed.

Let us consider an example. The command INSERT INTO titles (title, subtitle, langID) VALUES (?, ?, ?) will be sent to the server with mysql_prepare. Then, three parameters (two strings and an integer) will be declared in a MYSQL_BIND field and added to the command with mysql_bind_param. With the strings, a buffer will be used for data transfer; with the integer, on the other hand, a direct connection to a variable (here langID) will be made.

The command can now be executed as often as you like with mysql_execute after the parameters have been set:

// example c/prepare/main.c
#include <stdio.h>
#include <mysql.h>
int {main}(int argc, char *argv[]) { MYSQL *conn; // connection to MySQL server
MYSQL_STMT *stmt; // prepared statement
MYSQL_BIND bind[3]; // parameters
char *insert =
"INSERT INTO titles (title, subtitle, langID) VALUES (?, ?, ?)";
char titlebuf[256], subtitlebuf[256];
int langID;
// establish connection to MySQL
conn = mysql_init(NULL);
...
// prepare SQL command with three parameters
stmt = mysql_prepare(conn, insert, strlen(insert));
bind[0].buffer_type = FIELD_TYPE_STRING;
bind[0].buffer = titlebuf;
bind[0].buffer_length = 256;
bind[1].buffer_type = FIELD_TYPE_STRING;
bind[1].buffer = subtitlebuf;
bind[1].buffer_length = 256;
bind[2].buffer_type = FIELD_TYPE_LONG;
bind[2].buffer = (gptr) &langID;
mysql_bind_param(stmt, bind);
// execute SQL command
strcpy(titlebuf, "title1");
strcpy(subtitlebuf, "subtitle1");
langID=999;
mysql_execute(stmt);
// execute SQL command again with other parameters
strcpy(titlebuf, "title2");
strcpy(subtitlebuf, "subtitle2");
langID++;
mysql_execute(stmt);
// close connection
mysql_close(conn);
return 0; }


More New Stuff

Operators: There are three new operators: DIV for integer division, MOD for the modulus function (the remainder upon integer division), and SOUNDS LIKE for text comparison. WHERE SOUNDEX(a) = SOUNDEX(b) is equivalent to WHERE a SOUNDS LIKE b.

ROLLUP: With GROUP BY queries, the new extension WITH ROLLUP results in a partial sum being created for each group, as well as a total sum for all the results. The following example determines how many discussion contributions there are in the individual discussion groups of bigforum and in which years they were written. The NULL rows contain the partial sums and complete sum:

USE bigforum
SELECT forumID, YEAR(timest) AS year, COUNT(*)

FROM messages
WHERE forumID<=3
GROUP BY forumID, year
WITH ROLLUP

forumID | year | COUNT(*)
1 | 2000 | 11
1 | 2001 | 101
1 | 2002 7 | 158
1 | 2003 | 16
1 | [NULL] | 286
2 | 2000 | 7
2 | 2001 | 64
2 | 2002 | 27
2 | 2003 | 2
2 | [NULL] | 100
3 | 2000 | 2
3 | 2001 | 31
3 | 2002 | 46
3 | 2003 | 2
3 | [NULL] | 81
[NULL] | [NULL] | 467

Compression functions: COMPRESS(string) compresses the passed character string and returns as result a binary object that can be stored space-efficiently in a BLOB column. UNCOMPRESS(binarydata) undoes the compression. UNCOMPRESSED_LENGTH(binarydata) returns the length of the original character string.

Warnings and errors: With SHOW WARNINGS and SHOW ERRORS one can determine the warnings and errors triggered by the last-executed command. The variant forms SHOW COUNT(*) WARNINGS/ERRORS return the number of warnings/errors. Access to warnings is especially practical for the importation of text files with LOAD DATA.

Help: The new command HELP returns a brief help text to the specified functions. However, this command did not function in the version tested, so I cannot offer an example here.

Database directory: The new file db.opt, containing database options, is stored within database directories. (In the tested version, all that was in the file was the setting for the default character set for the database.)

Creating tables with identical layout: A new variant of CREATE TABLE simplifies the creation of new tables that are defined exactly like an existing table. (The new table is empty.)

CREATE TABLEnewtable LIKE oldtable

MySQL monitor: The program mysql recognizes two new options: --reconnect makes the program automatically attempt to recreate a broken connection; --disable-connect turns off this behavior. (The default is --reconnect.)


Client Programming

The libraries for client programming (C-API, PHP-API, Connector/ODBC, Connector/J) must be adapted for multibyte character sets. New versions of these APIs are already in development, but there was no documentation available at the time this book was completed. It is thus unclear, for example, how in the future a Unicode character string can be most easily stored by Perl, Java, VB, etc., in a MySQL table and then read out.

In the case of PHP, full support for the new MySQL functions is not expected until PHP 5, which was in beta testing as this text was written. However, some details are known. PHP will offer two MySQL APIs:

  • One of these APIs should be 100 percent compatible with the API of PHP 4.n described in this book.

  • However, the new MySQL functions will be supported only by the second MySQL API. Of course, many functions of this API can be used more or less as before, but there are many changes. The most apparent difference is that the function names begin with mysqli_ (instead of mysql_).


Incompatibilities

As far as I know, MySQL 4.1 is incompatible with MySQL 4.0 in at least two points:

  • PASSWORD returns longer and more robust encrypted character strings than previously. To encrypt strings as before, you must use OLD_PASSWORD.

    The stronger encryption of passwords affects the table mysql.user, in which access information for all MySQL users is stored. When the mysql database is taken over from an older MySQL version, the mysql database must be adapted with the script mysql_fix_privilege_tables. The column user.password will thereby be increased to 45 characters. New or changed passwords will automatically be encrypted with PASSWORD, while old passwords (with only 16 characters) will continue to be recognized and accepted.

  • Values from TIMESTAMP columns will now be formatted like DATETIME values, that is, by YYYY-MM-DD HH:MM:DD (instead of by YYYYMMDDHHMMDD).

If your program makes use of the old form, then simply add 0 to TIMESTAMP values (thus SELECT ts+0 FROM table).

It remains to be seen how much in the way of changes to client programming will be required as a result of the new character-set functions.


Overview of New and Changed Commands

The following list summarizes all SQL commands that are either completely new or whose syntax has been significantly altered or extended:

ALTER DATABASE (character set)

ALTER TABLE (character set)

CREATE DATABASE (character set)

CREATE TABLE (character set)

CREATE TABLE newtable LIKE existingtable DELETE (subSELECTs)

GROUP BY ... WITH ROLLUP

HELP

SELECT (subSELECTs)

SHOW CHARACTER SET

SHOW COLLATION

SHOW COLUMN TYPES

SHOW CREATE DATABASE

SHOW ERRORS, SHOW COUNT(*) ERRORS SHOW PRIVILEGES

SHOW TABLE TYPES

SHOW WARNINGS, SHOW COUNT(*) WARNINGS UPDATE (subSELECTs)

Additionally, there are many new or changed functions, of which we mention here only the most important. (In this list are missing, among others, countless new GIS functions.)

CHARSET

COMPRESS, UNCOMPRESS, UNCOMPRESSED_LENGTH CONVERT(s USING charset)

CRC32

PASSWORD (stronger encryption)

OLD_PASSWORD (encryption as before) VARIANCE

Moreover, there are new data types GEOMETRY and NCHAR, the index type SPATIAL INDEX, the constants TRUE and FALSE, and the operators MOD, DIV, and SOUNDS LIKE.


Appendix C: Example Files

The example files for this book are available as compressed files for download at the web site www.apress.com. To decompress them under Windows, you can use, for example, WinZip (http://www.winzip.com/), while under Unix/Linux, you use the command unzip.

Trying Out the Example Programs

Remark

All the example programs will work only if the databases described in this book are made available and if you change the user names and passwords in the code so that they are accepted by your MySQL installation.

Some of the examples assume at least MySQL 4.0.n, while most are compatible withMySQL 3.23.n.

PHP examples: All the PHP files assume at least PHP 4.1. Better would be PHP 4.2 or 4.3. You must change the login name and password in the code! This affects the following files:

general/*.php
myforum/myforumpassword.inc.php
mylibrary/mylibrarypassword.inc.php
pictures/connect.inc.php
vote_xxx/results.php

If you have problems in executing the example programs, please read the section "Error Checking" of Chapter 11.


Updates, Errata, Links, Discussion Forum

A collection of MySQL links, a discussion forum for this book, and any updates and errata that come to my attention can be found at my web site: http://www.kofler.cc.


Appendix D: Bibliography

  1. Sharon Bjeletich et al. Microsoft SQL Server 7.0 Unleashed. SAMS, 1999.

  2. Rich Bowen et al. Apache Server Unleashed. SAMS, 2000.

  3. Chris Date, Hugh Darwen. A Guide to the SQL Standard. Addison-Wesley Longman, 1998.

  4. Paul DuBois. MySQL, second edition. Sams, 2003.

  5. Paul DuBois. MySQL Cookbook. O'Reilly, 2002.

  6. W. J. Gilmore. A Programmer's Introduction to PHP 4.0. Apress, 2001.

  7. Cay S. Horstmann, Gary Cornell. Core Java 2, volume 2. Sun Microsystems Press/Prentice Hall, 1997.

  8. Michael Kofler. Definitive Guide to Excel VBA, second edition. Apress, 2003.

  9. Michael Kofler. Linux: Installation, Configuration, and Use, second edition. Addison-Wesley, 1999.

  10. Michael Kofler. Visual Basic Database Programming. Addison-Wesley, 2001.

  11. Laura Lemay. Teach Yourself Perl in 21 Days, third edition. SAMS, 1999.

  12. Robert Orfali, Dan Harkey, Jeri Edwards. Client/Server Survival Guide, third edition. John Wiley & Sons, 1999.

  13. Tobias Ratschiller, Till Gerken. Web Application Development with PHP 4.0. New Riders, 2000.

  14. David Sceppa. Microsoft ADO.NET (Core Reference). Microsoft Press, 2002.

  15. David Solomon et al. Microsoft SQL Server 6.5 Unleashed. SAMS, 1996.

  16. Carsten Thomsen. Database Programming with Visual Basic .NET. Apress, 2002.

  17. Luke Welling, Laura Thomson. PHP and MySQL Web Development. SAMS, 2001.

  18. Randy Yarger, George Reese, Tim King. MySQL & mSQL. O'Reilly, 1999.


Index

A

abbreviations, xviii
ABS, 693
Access, 584
#deleted error, 586
Access denied for user x to database y (error), 293, 298
access privileges, 262
changing with GRANT, 291, 669
changing with mysql_setpermission, 295
first aid, 267
limit with REVOKE, 678
to MySQL without a password, 271
restricting with DELETE, 267
restricting with REVOKE, 292
two-tiered security, 275
with UPDATE, 273
update from 3.23 to 4.0, 318
viewing, 292, 294
ACID, 246
ACL, 262
ACOS, 693
ActivePerl, 56
ActualSize property (ADO), 602
addBatch method (Java), 545
Add method (ADO.NET), 629
AddModule (Apache), 48
addslashes (PHP), 392, 398
addtypeAddtype (Apache), 48
administration
access privileges, 262
backups, 310
basic, 306
character set, 331
configuration files, 307, 707
database migration, 316
InnoDB, 357
inernational adaptation, 331
ISP, 369
logging, 335
MyISAM, 353
replication, 342
security, 262
text import/export, 320
user interface, 86
ADO, xviii, 579, 592
Unicode, 601
ADO.NET, 579, 616
AGAINST, 232
aggregate function, 700
alias
Apache, 42
SQL, 166
ALL, 679, 684
Allow (Apache), 43
AllowOverride (Apache), 43
All privilege, 275
alpha version, 11
ALTER, 658
ALTER DATABASE, 657
ALTER TABLE, 157
example, 226
ALTER TABLE TYPE, 245
ANALYSE, 676
ANALYZE TABLE, 353
AND (&&), 647
anonymous user, 280
ANSI SQL/92, 9
Apache
CGI configuration (Perl), 60
DBI module, 498
htaccess , 43
Linux installation, 40
mod-perl , 60
PHP configuration, 48
Windows installation, 39
AppendChunk method, 601
arithmetic
functions, 693
operators, 648
array_item (PHP example), 414
AS, 190, 680
example, 166
AsBinary, 774
ASCII, 696
ASCII import
phpMyAdmin, 113
ASIN, 693
AsText, 774
ATAN, ATAN2, 693
atomicity (ACID), 246
attributes
table columns, 663
@ (variables), 207
auto commit mode, 248
AUTO_INCREMENT, 139, 194, 663
ADO, 603
ADO.NET, 628
INSERT, UPDATE, 672
Java, 535
Perl, 499, 509
PHP, 384
AVG, 700


Index_B

Team LiB

Index

B

backslash (special characters), 643
backups, 191, 310
phpMyAdmin, 111
recreating a database, 313
restoring databases (InnoDB), 314
BACKUP TABLE , 192, 660
BadConversion exception (C), 575
BadNullConversion exception (C), 575
BadQuery exception (C), 571, 575
basedir (mysqld), 711
batch commands (Java), 544
batch mode (mysql), 329
BDB, 242
tables, 152
BDB, 664
BEGIN, 247, 660
BeginTransaction method (ADO.NET), 632
BENCHMARK, 701
Berkeley Databases, 242
beta version, 11
BETWEEN, 647
bigforum example database, 171
BIGINT, 138
BIN, 696
binary
AND (&), 647
comparison, 650
data, 144
C programming, 564
logging, 337, 346
OR (|), 647
BINARY
binary comparison, 202
binary search, 650
CHAR/VARCHAR, 143
bind-address (mysqld ), 718
bind_col (Perl), 502
bind_columns (Perl), 502
binlog-do-db (mysqld ), 714
binlog-ignore-db (mysqld), 714
bin2hex (PHP), 393
bit operators, 647
BIT_AND, 700
BIT_COUNT, 701
BIT_OR, 700
BLOB, xviii, 144
ADO, 601
ADO.NET, 627
Java, 546
max-allowed-packet (mysqld), 718
Perl, 508
PHP, 391
phpMyAdmin, 105
special characters, 643
writing to file, 684
books database (design), 123
buffer configuration, 366
build_categories_select (PHP example), 447
build_href (PHP example), 414
build_select_list (PHP example), 446
bulk insert, 320, 672
bulk_insert_buffer_size (mysqld), 367
ByteFX.Data provider (ADO.NET), 617, 621


Index_C

Team LiB

Index

C

C (programming language), 556
Can't connect to MySQL server (error), 297
Can't create table name (error), 257
Can't find any matching row (error), 293
C API, 556
syntax summary, 752
C++ API, 567
CASCADE, 255
CASE, 209
case sensitivity, 642
CAST, 694
categories table, 161, 165
delete category, 464
design optimization, 165
insert new categories, 462
CEILING, 693
CGI
configuration, 60
first test, 61
CHANGE MASTER TO, 348, 661
CHAR, 142, 696
character set
changing, 331
Connector/ODBC, 63
create new index, 356
HTML, 52
MySQL 4.1, 768
character string functions (SQL), 200
CHARACTER SET, 657, 661, 663, 664, 771
character-sets-dir (mysqld), 334
character strings, 142
syntax, 643
CHARSET, 771
charset attribute (HTML), 52
CHAR_LENGTH, 694
check sum, 696
CHECK_SUM, 664
CHECK TABLE , 353, 661
chmod via FTP, 61
CIPHER (GRANT), 669
Classic version, 17
client, 4
client license, 16
client-side cursor (ADO), 598
Clob class (Java), 546
close method (C++), 571
close method (Java), 530, 541
cmd.exe, 91
*.cnf files, 307, 707
COALESCE, 693, 701
Codd, E. F., 129
COLLATE, 657, 661, 663, 771
columns
attributes, 663
changing, 659
enabling access, 292
columns_priv table, 289
command interpreter (mysql), 87
commands out of sync error (C++), 573
COMMENT, 664
comments, 646
commercial MySQL license, 15
COMMIT, 247, 661
Commit method (ADO.NET), 632
commit method (Java), 544
comparison
functions, 693
operators, 648
compiled SQL commands, 775
compiling
C, 558
COMPRESS, 777
compressed MyISAM tables, 152
CONCAT, 166, 200, 694
conditions (SQL), 183, 680
*.conf file, 333
config.inc.php file, 54, 103
configuration files, 307, 707
connection
Java, 532
problems, 296
Connection class (ADO), 594
Connection class (C++), 571
Connection class (Java), 530
connection (DSN), 594
ConnectionString property (ADO), 594
CONNECTION_ID, 701
connect method (C++), 571
Connector/J, 525
JSP installation, 550
Connector/ODBC, 578
character set problem, 63
configuration (via DSN), 580
installation, 62
Connector/C++, 567
connect (Perl), 497
consistency (ACID), 246
console (mysqld-Start ), 29
constants, 653
CONV, 696
CONVERT, 694, 771
converter
Access MySQL, 588
MSSQL MySQL, 635
MySQL Access, 589
cookie authentication (phpMyAdmin), 105
cookies (PHP sessions), 472
CoreLab, 617, 623
COS, 693
COT, 693
COUNT, 180, 190, 700
Count property (ADO.NET), 629
CREATE DATABASE, 661
example, 156
CREATE FUNCTION, 662
CREATE INDEX, 158, 662
create listbox (PHP), 446
Create privilege, 662
CREATE TABLE, 156, 662
example, 226
from SELECT result, 665
LIKE oldtable, 666
from a SELECT result, 210
CREATE TABLE . . . SELECT, 197
Create Temporary Table privilege, 662
CROSS JOIN, 187
C#, 616
CSV files, 325
CURDATE, 697
cursor (ADO/ODBC)
client-side cursor), 598
dynamic cursor, 583
CURTIME, 697


Index_D

Team LiB

Index

D

daemon, 36
Data Control Language, 179
Data Definition Language, 177, 179
Data Manipulation Language, 177, 179
data record
alter, 195, 691
delete, 196, 666
insert, 193, 671
random selection, 229
search (SELECT), 679
data types, 137
data warehouse, 212
database
alter, 657
backups, 311
client/server system, 4
creating a new, 271, 661
delete, 667
enabling access, 291
glossary, 4
ISP backup, 369
migration, 316
model, 5
normalization, 123
object-oriented, 5
recreating, 313
relational, 4
renaming, 677
restoring (InnodB), 314
server, 4
upload to ISP, 369
DATABASE, 701
database design, 122
CREATE TABLE, 155
data types, 137
example, 158
indexes, 146
keys, 133
normal form, 123
optimization, 163, 172
referential integrity, 136
relations, 131
table types, 151
datadir (mysqld), 711
DataEnvironment (VB), 596
DataRow class (ADO.NET), 629
DataSet class (ADO.NET), 628
DataSource class (Java), 531
Data Source Name, 580
DataTable class (ADO.NET), 628
data types
reference, 653
date, 204
data type, 141
functions, 697
syntax, 645
DATE, 141, 204
DATE_ADD, 699
DATE_FORMAT, 141, 699
DATETIME, 141, 204
ADO, 601
DATE_ADD, 697
DATE_FORMAT, 697
DATE_SUB, 697
DAYNAME, 697
DAYOFMONTH, 697
DAYOFWEEK , 697
DAYOFYEAR, 697
DBD-Mysql , 57
DBD::mysql, 496
MySQL-specific methods, 509
*.db files, 154
DBI, 57
logging, 512
DBI, 496, 738
DBNull (ADO.NET), 627
db.opt file, 657, 777
dbProvider (ADO.NET), 617, 622
db table (access protection), 287
DBtools, 86
DCL, 179
DDL, 177, 179
deadlock, 252
DECIMAL, 140
ADO, 601
decimal numbers, 645
DECODE, 697
DEFAULT, 137, 663
INSERT, UPDATE , 672
default security settings, 265
default-character-set (mysqld), 333
default_charset (PHP), 52
default database
PHP, 379
SQL, 691
default-table-type (mysqld), 718
DEGREE, 693
DELAYED, 671
DELAY_KEY_WRITE, 146, 664
DELETE, 196, 666
FROM, 196, 666
LIMIT , 197, 666
linked tables, 196, 224, 666
ORDER BY, 197
USING, 666
delete first or last n records, 197
#deleted (Access), 586
delete-invalid-entries.pl example, 512
Delete method (ADO.NET), 629
deleteRow method (Java), 540
delete DELETE
invalid records, 512
demand mode (query cache), 369
DESC, 681
DESCRIBE, 667
design, see database design
Directory tag (Apache), 43
DISABLE KEYS, 659
disable-xxx (mysqld ), 710
disconnect (Perl), 498
discussion group (PHP), 467
display_errors (PHP), 52
DISTINCT, 186, 679
example, 168, 226
DIV, 647
DML, 177, 179
DO, 667
do (Perl), 498
example with parameters, 520
DOUBLE, 139
DriverManager class (Java), 529
DROP, 660
DROP DATABASE, 667
DROP FUNCTION, 667
DROP INDEX, 667
DROP TABLE, 667
DSN, 580
*.dsn files, 580
DSO, 47
DUMPFILE, 684
duplicates
searching for, 224
durability (ACID), 246
dynamic cursor (ADO/ODBC), 583
dynamic MyISAM tables, 151


Index_E

Team LiB

Index

E

echo (PHP), 439
efficient database design, 163, 172
eInfoDesign, 617, 622
ELT, 694
Emacs, 64
empty cache (FLUSH), 668
empty (PHP), 395
ENABLE KEYS, 659
enable-xxx (mysqld ), 710
enable-named-pipes (mysqld), 712
ENCLOSED BY, 328, 673, 683
ENCODE, 697
encode_entities (Perl), 508
encodeHTML method
Java, 552
ENCRYPT, 697
encryption, 697
ENUM, 144
determining character strings, PHP, 395
enumeration, 144
environment variables, 709
equality (=), 647
equality with NULL (<=>), 647
*.err file, 340
error
2002, 297
2003, 297
logging, 340
messages in other languages, 332
errors in establishing connections, 296
err (Perl), 511
errstr (Perl), 511
ESCAPED BY, 328, 673, 683
escapeHTML (Perl), 508
/etc/hosts file
Red Hat, 285
SuSE, 285
/etc/my.cnf file, 707
Excel, 590
exceptions (C), 571
exceptions database
design, 174
import/export, 323
sort order, 334
exclusive lock, 251
exec method (C++), 571
executeBatch method (Java), 545
Execute method (ADO), 604
ExecuteNonQuery method (ADO.NET), 624
execute (Perl), 499
executeQuery method (Java), 536
ExecuteReader method (ADO.NET), 624
ExecuteScalar method (ADO.NET), 624, 628
executeUpdate method (Java), 534
EXISTS , 217, 767
EXP, 693
EXPLAIN, 668
EXPLAIN SELECT, 149, 668
export
Access, 584
text files, 320
exportsql.txt converter, 588
EXPORT_SET, 694
extension_dir (PHP), 52
EXTRACT, 697


Index_F

Team LiB

Index

F

FALSE, 653
fetchall_arrayref (Perl), 504
fetch (Perl), 500
fetchrow_array (Perl), 500
fetchrow_arrayref (Perl), 504
fetchrow_hashref (Perl), 504
fetch_row method (C++), 572
FIELD, 694
FIELDS TERMINATED BY, 673, 683
example, 328
file DSN, 580
file transfer (HTML upload), 404
file transfer (upload ), 405
File privilege, 277
$_FILES field (PHP), 406
Fill method (ADO.NET), 628
FIND_IN_SET, 145, 694
find.php (example), 417
finish (Perl), 499
Firebird, 18
firewall, 303
FLOAT, 139
floating-point numbers, 139, 645
FLOOR, 693
FLUSH, 668
FLUSH LOGS, 339
FLUSH PRIVILEGES, 287
foreign key, 125, 135
integrity rules, 255
foreign key constraints
deleting, 258
InnoDB, 255
FOREIGN KEY, 136, 255
foreign_key_checks variable, 259, 365
A foreign key constraint fails (error), 257
form (HTML), 396
file transfer (upload), 405
form evaluation (PHP), 396
file transfer (upload), 406
FORMAT, 696
form (HTML)
create listbox (PHP), 446
hierarchical listbox (PHP), 447
forumread.php example, 478
forumlist.php example, 477
forumlogin.php example, 473
forumlogout.php example, 492
forummessage.php example, 484
forumnewlogin.php example, 475
forumthread.php example, 486
forumwrite.php example, 487
FOR UPDATE, 251
forward-only ResultSets (Java), 539
FOUND_ROWS, 181, 701
*.frm files, 154
FROM, 179
DELETE, 196, 666
FROM_UNIXTIME, 392, 506
FROM_DAYS, 697
FROM_UNIXTIME, 697
ft_min_word_len (mysqld), 234
FULLTEXT, 232, 663
full-text
Boolean search expressions, 232
index, 148, 231
search, 231
function
GROUP BY, 700
arithmetic, 693
comparison, 693
date and time, 697
string, 694


Index_G

Team LiB

Index

G

gamma version, 11
gap lock, 251, 254
gcc, 558
Gemini, 243
isolation degree for transactions, 252
tables, 153
general query log, 340
geographic data, 773
geometric data, 773
GEOMETRY, 773
GEOMETRYCOLLECTION, 773
GeomFromText, 774
GeomFromWBK, 774
german1 character set, 335
getAutoCommit method (Java), 544
getBinaryStream method (Java), 546
getBlob/Clob method (Java), 546
getBytes method (Java), 546
getCharacterStream method (Java), 546
GetChunk method (ADO), 602
getColumnCount method (Java), 538
getColumnName method (Java), 538
getColumnType method (Java), 538
getConnection method (Java), 529
getGeneratedKeys method (Java), 535
getLastInsertID method (Java), 535
GET_LOCK, 702
get_magic_quotes_gpc (PHP), 399
get_magic_quotes_runtime (PHP), 399
getMetaData method (Java), 538
getXxx methods (Java), 537
GIS, 773
GLOBAL, 652, 684
global privileges, 277
global variables, 652
PHP, 376
$GLOBALS field (PHP), 376
glossary, 4, 761
*.gmd files, 154
*.gmi files, 154
GPL, 14
GRANT, 290
example, 291
syntax, 669
Grant Option privilege, 276
grant tables, 278
GREATEST, 693
GROUP BY, 189
functions, 700
GROUP BY WITH ROLLUP, 777
GUI, 86


Index_H

Team LiB

Index

H

HANDLER, 670
HAVING, 184
header (PHP), 407
HEAP tables, 153, 664
maximum size, 713
HELP, 671
HEX, 696
hexadecimal numbers
BLOB, 643
PHP, 393
SQL, 645
HexCode function (VB), 604
HexCodeStr function (VB), 605
hierarchical listbox (PHP), 447
hierarchy
example (myforum database), 171
logical sequence, 165
realization with tables, 132
HIGH_PRIORITY, 679
host name, 263
user table, 280
with or without domain name, 283
hostname bin.index file, 338
hostname bin.nnn file, 338
hostname.log file, 340
hostname-relay files, 350
hostname-slow.log file, 341
hostname (Unix/Linux command), 285
Host n.n.n.n is not allowed to connect ...(error), 297
host table (access protection), 288
host (Unix/Linux command), 285
hot backup, 10
HOUR, 697
*.htaccess file (Apache), 43
htdocs directory, 42
HTML
character set, 52
HTML::Entities module, 508
htmlentities (PHP), 387, 394
versus htmlspecialchars, 439
htmlspecialchars (PHP), 398
versus htmlentities, 439
HTML tables (mysql ), 330
htpasswd, 44
HTTP_COOKIE_VARS, 400
httpd.conf file, 40
HTTP_GET_VARS, 400
HTTP_POST_VARS, 400


Index_I

Team LiB

Index

I

ib_arch_log_n file, 364
ib_logfilen, 361
IDENTITY, 701
IF, 209, 212, 693
example, 200
IFNULL, 693
IGNORE, 671
IGNORE . . . LINES, 674
IGNORE (SELECT), 665
Images, storing and displaying (PHP), 404
import
Access, 584
CSV files, 325
mysqlimport, 326
phpMyAdmin, 113
text files, 320, 672
IN, 217, 680, 767
in, 184
IN2IN, 647
IN BOOLEAN MODE, 232
index, 6
analyze distribution, 660
create new (other character set), 356
creating, 158, 662
delete, 667
display index list, 688
full-text, 231
fundamentals, 146
primary, 134, 147
temporarily deactivate, 659
UNIQUE, 147
INDEX, 663
Indexes (Apache), 43
!= (inequality), 647
INET_ATON, 696
INET_NTOA, 696
INFILE , 672
*.ini files, 307, 707
INNER JOIN, 187
InnoDB, 240
administration, 357
changing the character set, 334
deadlock, 252
foreign key constraints, 255
integrity rules, 255
locking, 250
long transactions, 248
options, 716, 717
replication, 351
restoring tables, 314
tables, 152
tablespace, 357
timeout, 252
transactions, 245
logging, 361
tuning, 364
INNODB, 664
innodb_additonal_mem_pool_size ( mysqld configuration, 364
innodb_buffer_pool_size (mysqld), 364
innodb_data_file_path (mysqld ), 358
innodb_data_home (mysqld), 358
innodb_flush_method (mysqld ), 363
innodb_log_archive (mysqld), 363
innodb_flush_log_at_trx_commit (mysqld), 363
innodb_log_buffer_size (mysqld configuration), 364
innodb_log_files_in_group (mysqld ), 362
innodb_log_file_size (mysqld ), 362
innodb_log_group_home (mysqld ), 362
INSERT, 193, 694
default value, 672
example, 157
syntax, 671
insert_id method (C++), 571
INSERT INTO SELECT
example, 192, 211, 226
syntax, 671
insert_new_category, 462
insertRow method (Java), 540
IN SHARE MODE, 250
installation, 23
MySQL under Linux, 29
MySQL under Windows, 24
install-manual (mysqld) , 29
install (mysqld), 29
INSTR, 694
INT, 138
integer types, 138
integrity rules
conditions, 257
deactivating, 259
deleting, 258
restoring tables, 314
Interbase, 18
international adaptation, 331
INTERSECT, 189
INTERVAL, 693
invalid records, searching for, 221
IO_THREAD, 690
ISAM, 664
ISAM tables, 152
isAutoIncrement method (Java), 538
IsDBNull method (ADO.NET), 627
IS_FREE_LOCK, 701
ISNULL, 184, 648, 693
isNullable method (Java), 538
is_null method (C++), 573
is_numeric (PHP), 395
ISO-8859-1 character set, 331
isolation (ACID), 246
isolation level, 252
ISP, xviii
administering MySQL, 64
administration, 369
backup, 371
database upload, 371
phpMyAdmin configuration, 103
isset (PHP), 395
is_string (PHP), 395
ISSUER (GRANT), 669
is_uploaded_file (PHP), 406


Index_J

Team LiB

Index

J

Jakarta, see Tomcat
Java, 523
problems with establishing a connection, 532
javac, 525
java.sql package, 528
javax.sql package, 531
JDBC, 525
syntax reference, 745
JDK, 524
JOIN, 184
join_buffer_size (mysqld), 367
JSP, 548
JSTL (JSP Standard Tag Library), 551


Index_K

Team LiB

Index

K

key, 6
KEY, 663
key_buffer_size (mysqld), 366
KILL, 672


Index_L

Team LiB

Index

L

LAMP, xiii
language (mysqld), 332
LastInsertedID function (VB), 603
LAST_INSERT_ID, 139, 194, 701
ADO.NET, 628
last_name_first/last (PHP example), 445
latin1 character set, 331, 332, 334
LCASE, 694
ldd, 560
LEAST, 693
LEFT, 694
LEFT JOIN, 185, 187
example, 426
LENGTH, 200, 694
libmysqlclient, 33
C programming, 556
libmySQL.dll library, 612
license (GPL), 14
LIKE, 201
LIMIT, 181, 682
DELETE, 197
FOUND_ROWS, 181
LINES TERMINATED BY, 673, 683
LINESTRING, 773
linked tables, 131
Access, 586
phpMyAdmin, 116
linking
C, 558
LOAD DATA, 323, 672
logging, 338
replication, 344
LOAD DATA FROM MASTER, 349, 675
LOAD DATA LOCAL, 673
load_defaults (C), 562
LOAD_FILE, 701
LoadFromFile method (ADO), 602
LoadModule (Apache), 48
LOAD TABLE FROM MASTER, 675
LOCAL, 652, 673
localhost, 42
host name, 280
problems under Unix/Linux, 283
problems under mysqladmin (Windows), 269
local-infile (mysqld ), 712
LOCATE, 694
LOCK example, 462
locking, 10, 675
locks, 702
LOCK TABLE, 675
LOG, LOG10, 693
log-bin (mysqld), 346
log-error (mysqld), 340
*.log files, 340
logging, 335
administration of logging files, 342
binary format, 337
errors, 340
InnoDB, 361
logins, 340
logging file
delete, 677
logins
logging, 340
log-long-format, 341
log (mysqld, 340
log-slow-queries, 341
log-update (mysqld ), 337
LONGBLOB, 144
long_query_time (mysqld), 341
LongReadLen (Perl), 508
LONGTEXT, 143
LongTruncOK (Perl), 740
loose-xxx (mysqld), 710
lost connection to MySQL (error), 31
LOWER, 694
lower_case_table_names (mysqld ), 642
LOW_PRIORITY, 671
LPAD , 694
LTRIM, 694


Index_M

Team LiB

Index

M

magic quotes (PHP), 398
Makefile
C projects, 558
C++ projects, 570
MAKE_SET, 694
master.info file, 350, 689
change settings, 661
master-xxx (mysqld), 715
master/slave replication, 343
master system
setup, 345
MATCH, 232
MAX, 700
Max version, 17
max-allowed-packet (mysqld), 718
max_binlog_size (mysqld), 339
max_connections column (user table), 282
max_connections (mysqld ), 367, 383
max_execution_time (PHP), 408
max_heap_table_size (mysqld ), 713
maximum-xxx (mysqld ), 710
max_input_time (PHP), 408
max_questions column (user table), 282
max_updates column (user table), 282
MD5, 696, 697
MDAC, 579
MEDIUMBLOB, 144
MEDIUMINT, 138
MEDIUMTEXT, 143
memory management, 366
memory_limit (PHP), 408
MERGE, 664
MERGE tables, 153
metadata to the ResultSet (Java), 538
<meta> tag, 52
Microsoft SQL Server, MySQL converter, 635
MID, 694
migration, 316
MIN, 700
MINUS, 189
MINUTE, 697
MM.MySQL, 526
MOD, 647, 693
model (database), 5
MODIFY, 660
mod-perl, 60
monitor, 87
MONTHNAME, 697
moveToInsertRow method (Java), 540
Mozilla (Cache problems), 50
mSQL, 7
MS Query, 590
mssql2mysql, 636
MULTILINESTRING, 773
MULTIPOINT, 773
MULTIPOLYGON, 773
MultiViews (Apache), 43
mybackup, 313
MyCC, 97
my.cnf file, 707
*.MYD files, 154, 729
myforum database
design, 170
PHP code, 467
*.MYI files, 154, 729
my.ini file, 707
MYISAM, 664
myisamchk, 353
syntax, 729
myisam-recover (mysqld), 712
myisampack, 356
syntax, 731
MyISAM tables, 151
compressing, 356
create new index, 356
maintenance, 353
repairing, 356
shrinking, 355
mylibrary database, 158
MyVbQL example, 614
ADO Visual Basic example, 605
CGI Perl example, 514
deleting invalid records, 512
Perl example, 512
PHP example, 414
mylibrary-find.pl example, 515
mylibraryinno, example database, 259
mylibraryodbc database, 159, 594
mylibrary-simpleinput.pl (example), 518
MyODBC, see Connector/ODBC
my_print_defaults, 709
MySQL
Classic version, 17
commercial license, 15
installation, 23
license, 14
Linux installation, 29
Max version, 17
memory management, 366
overview, 6
Pro version, 17
service (Windows NT/2000/XP), 24, 28
Standard version, 17
testing the installation, 35
tuning, 365
update, 36
update from 3.23 to 4.0, 318
using with ISP, 64
Windows installation, 24
mysql, 647
MySQL Control Center, 97
MySQL Navigator, 86
MySQL versions
Standard, Max, Classic, Pro, 17
Windows, 28
mysqlaccess, 294
mysqladmin, 293, 309
problems under Windows, 269, 293
syntax, 723
mysql_affected_rows (C), 562
mysql_affected_rows (PHP), 384, 391, 736
mysql_bind_param (C), 776
mysqlbinlog, 337
mysqlbug, 728
mysql_change_db (C), 754
mysql_change_user (C), 754
mysql_change_user (PHP), 734
mysql_characterset_name (C), 755
mysql_client_encoding (PHP), 735
mysql_close (C), 561
mysql_close (PHP), 734
MySQLCommand class (ByteFX provider), 626
mysql_connect (PHP), 378
MySQLConnection class (ByteFX provider), 621
MySQLConnection class (dbProvider ), 622
mySqlConnection class (MySQLDirect .NET), 623
mysql_connect (PHP), 734
mysql_convert_table_format, 245
mysql_create_db (PHP), 734
mysqld, 30, 36, 70
basic administration, 307
configuration, 711
configuration with WinMySQLadmin, 94
option reference, 711
starting/stopping, 307
mysql database, 277
update from 3.23 to 4.0, 318
mysql_data_seek (C), 757
mysql_data_seek (PHP), 736
mysql_db_query (PHP), 735
MySQLDirect .NET provider (ADO.NET), 617, 623
MySQLDriverCS provider (ADO.NET), 617
mysql_drop_db (PHP), 734
mysqld_safe, 718
mysqldump, 311, 329
creating a backup, 192
example, 371
syntax, 724
mysql_errno (C), 564
mysql_errno (PHP), 734
mysql_error (C), 564
mysql_error (PHP), 734
mysql_escape_string (PHP), 393
mysql_execute (C), 776
mysql_fetch_array (PHP), 386, 736
mysql_fetch_field (PHP), 387, 737
mysql_fetch_length (PHP), 737
mysql_fetch_lengths (C), 565
mysql_fetch_object (PHP), 386, 736
mysql_fetch_row (C), 563
mysql_fetch_row (PHP), 386, 736
mysql_field_flags (PHP), 387, 737
mysql_field_len (PHP), 386, 737
mysql_field_name (PHP), 386, 737
mysql_field_table (PHP), 386, 737
mysql_field_type (PHP), 386, 737
mysql_fix_extensions, 720
mysql_fix_privileges, 719
mysql_fix_privilege_tables, 318, 319
mysql_free_result (C), 563
mysql_free_result (PHP), 385, 736
mysql_get_client_info (C), 755
mysql_get_host_info (C), 755
mysql_get_xxx_info (PHP), 735
mysql_get_proto_info (C), 755
mysql_get_server_info (C), 755
mysqlhotcopy, 315
mysqlimport, 326
syntax, 727
mysql_info (PHP), 391
mysql_init (C), 561
mysql_insert_id (C), 562
mysql_insertid (Perl), 499, 509
mysql_insert_id (PHP), 384, 390, 736
mysql_install_db , 283, 719
mysql_install_db, 266
mysql_kill (C), 755
mysql_list_dbs (PHP), 734
mysql_list_fields (PHP), 734
mysql_list_tables (PHP), 734
mysql (monitor), 87
batch mode, 329
character set problems (DOS), 92
configuration file, 707
generating HTML tables, 330
interactive operation, 88
MySQL 4.1, 778
reading backup files, 313
syntax, 720
tips, Unix/Linux, 90
tips, Windows, 91
MySQLNet provider (ADO.NET), 617
mysql_num_fields (PHP), 384, 736
mysql_num_rows (C), 757
mysql_num_rows (PHP), 384, 736
mysql_options (C), 561
mysql_pconnect (PHP), 382, 734
mysql_ping (C), 754
mysql_prepare (C), 776
mysql_query (C), 562
mysql_query (PHP), 383, 735
mysql_query_test (PHP example), 403
mysql_query_unbuffered (PHP), 735
mysql_read_default_file (Perl), 497
mysql_read_default_group (Perl), 497
mysql_real_connect (C), 561
mysql_real_escape_string (PHP), 393
mysql_real_query (C), 565
MYSQL_RES (C), 563
mysql_result (PHP), 386, 736
mysql_row_seek (C), 757
mysql_row_tell (C), 757
mysql_select (PHP), 734
MySQL server
configuration files, 707
logging, 335
restarting, 307
mysql_setpermission, 295
mysqlshow, 308
syntax, 728
mysql_stat (C), 755
mysql_stat (PHP), 735
mysql_store_result (C), 563
mysql_store_result (Perl), 510
mysql_thread_id (C), 755
mysql_thread_id (PHP), 735
MysqlTool, 86
mysql_unbuffered_query (PHP), 389
mysql_use_result (C), 563
mysql_use_result (Perl), 510
MySQL versions
alpha, beta, gamma, . . . , 11
version numbers, 11
mysql_real_escape_string (C), 564
my_ulonglong (C), 562
myVbQL.dll library, 613
MyVbQL library, 612


Index_N

Team LiB

Index

N

named pipes, 300
named pipes (mysqld), 712
names
storing, 160
NATURAL JOIN, 187
NCHAR, 772
nested queries, 216, 767
net (Windows command), 29
NewRow method (ADO.NET), 629
next key lock, 251, 254
next method (Java), 536
n : m relation, 133
\N (NULL) in import/export, 322
NO ACTION, 255
nomenclature (databases, tables, columns, etc.), 642
normal form, 123, 124
bringing a table into, 225
normalization, 123
versus efficiency, 163, 172
! (NOT), 647
notee (mysql), 722
NOT IN, 647
NOT NULL, 135, 137, 663
NOW, 697
NULL , 393, 395, 663
ADO, 601
ADO.NET, 627
C programming, 563
column definition, 137
comparis on, 184
import/export, 322
Java, 537
operators, 648
Perl, 507, 508
PHP, 391
numbers, 645
NUMERIC, 140
num_or_null (PHP example), 444
Nusphere, 153


Index_O

Team LiB

Index

O

object privileges, 277
object-oriented database, 5
object names, 642
OCT, 696
ODBC, xviii, 578
installation, 62
ODBC data provider (ADO.NET), 616, 618
OdbcCommand class (ADO.NET), 619
OdbcConnection class (ADO.NET), 619
OdbcDataAdapter class (ADO.NET), 628
OdbcDataReader class (ADO.NET), 619, 626
OdbcException class (ADO.NET), 624
OdbcCommand class (ADO.NET), 624
OdbcTransaction class (ADO.NET), 632
OLAP, 212
old-passwords (mysqld), 712
BOLE-DB, 579
OLE-DB
driver for MySQL, 592
ON DELETE, 255
one-to-many relations, 132
one-to-one relations, 131
ON UPDATE, 255
open source, 14
OpenGIS, 773
Open method (ADO), 598
operators, 647
optimization
database design, 163, 172
query, 149
OPTIMIZE TABLE, 353, 676
options of MySQL tools, 705
OR (||), 647
ORD, 696
Order (Apache), 43
ORDER BY, 182, 681
DELETE, 197, 666
RAND, 183, 229
UPDATE , 195, 691
OUTER JOIN, 187
OUTFILE, 682


Index_P

Team LiB

Index

P

PACK_KEYS, 664
paged display of search results, 417
partitions, hard drive (InnoDB), 358
password, 263
changing, 293
forgotten root, 273
setting the root password, 267
user table, 280
PASSWORD, 697
MySQL 4.1, 778
pattern matching, 201, 649
OdbcCommandBuilder class (ADO.NET), 629
pdf files
creating with phpMyAdmin, 119
PEAR, 377
performance, replication, 343
Perl, 495
backups at an ISP, 372
CGI configuration (Apache), 60
database access, 497
DBD::mysql, 496
DBI, 496
DBI reference, 738
error-handling, 510
evaluating lists of records, 500
executing SQL commands, 498
installation, 56
persistent connection, 498
upload to ISP, 372
persistent connection
Perl, 498
PHP, 382
PGP
magic quotes, 398
PHP, 46
API reference, 734
BLOBs, 391
CGI versus module variant, 47
compatibility problems, 376
configuration, 51
error checking, 403
executing SQL commands, 383
extensions (extensions), 52
functions, see mysql
NULL , 391
persistent connection, 382
random server selection (replication), 352
upload to ISP, 371
Windows installation, 47
backup at an ISP, 371
*.php files, xvi, 48
php.ini file
basic configuration, 51
configuration for file transfer, 408
magic quotes, 398
phpMyAdmin, 101
additional functions, 115
administration at an ISP, 370
administrative tasks, 114
backups, 111
config.inc.php file, 54
config.inc.php file, 103
configuration options, 105
database design, 108
database uploading, 113
executing SQL commands, 111
insert, display, edit table data, 109
installation, 53
installation variants, 103
managing relations, 116
PI, 693
ping (mysqladmin), 309
Pivot Table, 212
*.pl files, 57
POINT , 773
POLYGON, 773
port 3306, 299, 303, 530
POSITION, 694
POST field (PHP), 396
PostgreSQL, 18
post_max_size (PHP), 408
POW, 693
POWER, 693
ppm (Perl package manager), 57
precompiled SQL commands, 775
prepared statements, 775
PreparedStatement class (Java), 541
prepare (Perl), 499
prepareStatment method (Java), 541
primary index, 6, 124, 134
primary key, 6, 124, 134, 147
PRIMARY KEY, 134, 663
PrintError (Perl), 511
privileges
global versus object, 277
list of all privileges, 275
user table, 281
Pro version, 17
PROCEDURE, 676
PROCESSLIST, 688
Process privilege, 277
production version, 11
programming (SQL), 207
programming techniques, 378
Prompt (ODBC option), 595
PURGE MASTER LOGS, 677
purge method (C++), 573


Index_Q

Team LiB

Index

Q

qq (Perl), 506
QT, 568
QUARTER, 697
queries, 6
execution analysis, 668
logging slow queries, 341
optimization, 149
SELECT introduction, 179
SELECT syntax, 679
SQL conditions, 183
query
nested, 216, 767
query cache, 368
query_cache_limit (mysqld), 368
query_cache_size (mysqld ), 368
query_cache_type (mysqld ), 368
Query class (C++), 572
QUICK (DELETE), 666
quote (Perl), 506
Quote (Visual Basic), 604
quoting (SQL), 643


Index_R

Team LiB

Index

R

RADIANS, 693
RaiseError (Perl), 510
RAND, 229, 693
replication, 344
random sort order, 183
random sorting, 229
random numbers, 229
RDBMS, 4
read_buffer_size (mysqld), 367
READ COMMITED, 252
readline library, 91
READ LOCAL LOCK, 675
READ LOCK, 675
Read method (ADO.NET), 626
read_rnd_buffer_size (mysqld), 367
READ UNCOMMITED, 252
REAL, 139
record, 5
record_buffer (mysqld ), 367
Recordset class (ADO), 598
testing properties, 600
recursive relations, 132
Red Hat
localhost problems, 285
MySQL installation, 30
REFERENCES, 136, 255, 663
referential integrity, 136
InnoDB, 255
REGEXP
examples, 201
syntax, 649
register_globals (PHP), 377
regular expression (REGEXP), 201, 649
rehash (mysql ), 722
relational database, 4
relations, 131, 132
in phpMyAdmin, 116
relay files, 350
RELEASE_LOCK, 702
remove (mysqld ), 29
RENAME AS, 660
RENAME TABLE, 677
REPAIR TABLE, 353, 677
REPEAT, 694
REPEATABLE READ, 252
REPLACE, 203, 677, 694
SELECT option, 665
replicate-xxx (mysqld ), 715
replication, 342
change slave configuration, 661
client programming, 352
delete logging files, 677
InnoDB tables, 351
logging interrupt, 685
master status, 688
master system, 345
reinitialize slave, 677, 678
determine slave list, 689
slave status, 689
slave system, 347
Replication Slave privilege, 345
REQUIRE (GRANT ), 669
RESET MASTER, 677
RESET QUERY CACHE, 369, 678
RESET SLAVE, 678
RESTORE TABLE, 192, 678
RESTRICT, 255
resultclasscResult class (C++), 572
ResultSet class (Java), 529, 536
forward-only, 539
variable ResultSets, 540
ResultSetMetaData class (Java), 538
ResUse class (C++), 572
REVERSE, 694
REVOKE, 290
syntax, 678
RLIKE, 201, 649
ROLLBACK, 247, 678
Rollback method (ADO.NET), 632
rollback method (Java), 544
ROLLBACK TO SAVEPOINT, 248, 682
ROLLUP summation, 777
root password
forgotten, 273
setting, 267
ROUND, 215, 693
Row class (C++), 572
rows method (C++), 572
rows (Perl), 502, 509
RPAD, 694
R-tree index, 774
RTRIM, 694


Index_S

Team LiB

Index

S

safe_mysqld, 718
safe-user-create (mysqld), 712
SAP DB, 18
SAPI, 47
SAVEPOINT, 248, 682
SaveToFile method (ADO), 602
ScriptInterpreterSource, 60
SECOND, 697
SEC_TO_TIME, 206, 697
security, 262, 669
recreating the default state (Unix/Linux), 266
default settings, 265
example, 291
first aid, 267
known risks, 302
operating-system level, 301
replication, 343
securing the installation, 267
two-tiered control, 275
update from 3.23 to 4.0, 318
SELECT , 668, 679, 684
displaying paged results, 417
example, 179
EXPLAIN, 668
SQL_CALC_FOUND_ROWS, 181
store result in new table, 665
storing a result in another table, 210
subSELECT, 216, 767
UNION, 188
variable assignment, 208
selectall_arrayref (Perl), 505
SELECT FOR UPDATE , 251
<select> (hierarchical listbox), 447
<select> (HTML listbox), 446
SELECT INTO OUTFILE, 327, 682
selection conditions (SQL), 680
SELECT
LOCK IN SHARE MODE, 250
selectrow_array (Perl), 502
example, 520
semicolon: mysql, 647
SERIAL, 655
server, 4, see also MySQL server, mysqld
tuning (optimization), 365
server-id (mysqld), 346
SESSION, 652, 684
Sessions (PHP), 472
session_start (PHP), 472
SESSION_USER, 701
SET, 145, 207
change MySQL options, 685
determining character strings (PHP), 395
determining character strings (Perl), 508
system variables, 684
user variable, 684
SET AUTOCOMMIT, 248, 685
setAutoCommit method (Java), 544
setBinaryStream method (Java), 546
setBlob/Clob method (Java), 546
setBytes method (Java), 546
setDatabaseName method (Java), 531
set_magic_quotes_runtime (PHP), 399
SET PASSWORD, 685
setServerName method (Java), 531
SET SQL_LOG_BIN , 685
SET TRANSACTION ISOLATION LEVEL, 252, 685
setURL method (Java), 531
setXxx methods (Java), 541
SHA, SHA1, 696
shared lock, 251
SHARE MODE, 250
SHOW BINLOG EVENTS, 686
SHOW CHARACTER SET, 769
SHOW CHARACTER SET, 686
SHOW COLLATION, 686
SHOW COLUMNS, 687
SHOW COLUMN TYPES, 686
SHOW CREATE DATABASE, 687
SHOW CREATE TABLE, 687
SHOW DATABASES, 687
SHOW ERRORS, 687
SHOW FIELDS, 687
SHOW GRANTS, 688
SHOW INDEX, 688
SHOW INNODB STATUS , 688
SHOW LOGS, 688
SHOW MASTER LOGS, 688
SHOW MASTER STATUS, 688
SHOW PRIVILEGES, 688
SHOW PROCESSLIST, 688
SHOW SLAVE HOSTS, 689
SHOW SLAVE STATUS, 689
SHOW STATUS, 689
SHOW TABLES, 689
SHOW TABLE STATUS, 689
SHOW TABLE TYPES, 689
SHOW VARIABLES, 651
SHOW VARIABLES, 690
SHOW WARNINGS, 690
SID (PHP), 472
SIGN, 693
SIN, 693
size method (C++), 572
skip-grant-tables (mysqld), 273, 301, 712
skip-host-cache (mysqld ), 301, 712
skip-xxx (mysqld), 710
skip-name-resolve (mysqld ), 712
skip-networking (mysqld), 297, 712
skip-show-database (mysqld ), 712
SLAVE STOP, 690
slave_compressed_protocol (mysqld), 715
slave-skip-errors (mysqld), 715
SLAVE START, 690
slave system setup, 347
slow query log, 341
SMALLINT, 138
snapshot, 346
socket file, 284, 297
socket (mysqld ), 705, 711
sort order
MySQL 4.1, 768
sort_buffer (mysqld), 367
sorting, 182
random order, 183, 229
sort order
setting, 333
SOUNDEX, 455, 696
source (mysql ), 722
SPACE, 694
example, 166
SPATIAL INDEX, 663, 773
specialized SQL structures (C++), 575
speed
optimization (InnoDB), 364
replication, 343
SQL, 6, 177
character strings, 200
command reference, 657
command overview, 655
function reference, 692
operators, 647
phpMyAdmin, 111
programming, 207
syntax, 642
SQL/92, 9
SQL_BIG_RESULT, 680
SQL_BUFFER_RESULT, 679
SQL_CACHE, 679
SQL_CALC_FOUND_ROWS, 181, 679
sql_create_basic macro (C++), 575
SQL_LOG_BIN, 685
SQL_LOB_BIN variable, 351
SQL_NO_CACHE, 679
SQL Server (Microsoft), MySQL converter, 635
SQL_SMALL_RESULT, 680
sql_str (PHP example), 444
<sql:xxx> tags
example, 553
reference, 750
SQL_THREAD, 690
SQLyog, 86
SQRT, 693
SSL
encryption, 282
GRANT, 669
SSQL (C++), 575
standalone (mysqld-Start), 29
Standard version, 17
START TRANSACTION, 247, 690
Statement class (Java), 529, 534
static MyISAM tables, 151
STATUS, 689
status (mysql), 90
STD, 700
STDDEV, 700
stored procedure, 10
storein method (C++), 576
store method (C++), 572
STRAIGHT_JOIN, 187, 680
STRCMP, 693
Stream class (ADO), 602
streaming ResultSet (Java), 539
string
conversion, 696
functions, 694
stripslashes (PHP), 399
strmov (C), 565
str_or_null (PHP Example), 444
structure, see database design
Structured Query Language, 6
SUBJECT (GRANT ), 669
subSELECT, 9, 216, 767
SUBSTRING, 200, 694
SUM, 212, 700
Superglobal fields (PHP), 376
Super privilege, 277
support, 18
surname (dealing with names), 160
SuSE
localhost problems, 285
MySQL installation, 31
system security, 301
system variable (@@), 651, 684, 690
SYSTEM_USER, 701
system variable (@@), 709
system DSN, 580


Index_T

Team LiB

Index

T

table
backup, 192, 660
changing type, 245, 658, 660
compressing, 356
copying, 210, 665
creating, 156, 662
creating a new index, 356
deleting, 667
determining internal errors, 661
editing the design, 157
enabling access, 292
exporting, 320
file, 154
importing, 320
with Access, 584
with phpMyAdmin, 113
locking, 675
optimizing, 676
options, 664
renaming, 660, 677
with phpMyAdmin, 114
repairing, 356
repairing a file, 353, 677
restoring, 678
shrinking, 355
type, 151
table_cache (mysqld), 367
tablespace, 244
administration, 357
tables_priv table, 289
<taglib> entries (web.xml), 551
TAN, 693
TCP/IP connection to local MySQL server, 283
tee (mysql), 90, 722
TEMPORARY, 662
temporary table, 153, 662
for complex DELETE commands, 229
in persistent PHP connections, 383
in place of subSELECT, 218
test database, 266
setting in the db table, 288
TEXT, 143
text files
table import, 320, 672
thread
discussion thread, 171
display, 688
display discussion thread, 478
terminate, 672
time, 204
data type, 141
MySQL syntax, 645
SQL functions, 697
syntax, 645
TIMESTAMP, 141
ADO, 601
INSERT, UPDATE, 672
MySQL 4.1, 778
PHP, 392, 394
Unix, 697
avoiding changes with UPDATE, 207
timestamps
Perl, 506, 507
TIME_TO_SEC, 697
TIME_FORMAT, 697
TINYBLOB, 144
TINYINT, 138
TINYTEXT, 143
tmp_table_size (mysqld ), 367
Tomcat
installation (SuSE), 548
TO_DAYS, 697
trace (Perl), 512
transaction, 8, 245
ADO.NET, 632
isolation level, 252
Java, 544
logging, 361
replication, 351
trigger, 10
TRIM, 694
TRUE, 653
TRUNCATE, 690, 693
tuning, 365
InnoDB, 364
persistent connections, 383
tx_isolation variable, 254
TYPE, 664


Index_U

Team LiB

Index

U

UCASE, 694
UDFs, 662
UNCOMPRESS, 777
UNCOMPRESSED_LENGTH, 777
Unicode, 11
ADO, 601
ADO.NET, 627
Java, 542
MySQL 4.1, 768
UCS2, UTF8, UTF16, 769
Visual Basic, 601, 605
UNION, 188, 684
UNIQUE, 147, 663
unique_checks variable, 364
UNIX_TIMESTAMP, 205, 394, 507, 697
UNLOCK TABLES, 691
UPDATE, 195
default value, 672
DELETE, 195
LIMIT, 195, 691
ORDER, 691
ORDER BY, 195
several tables, 196, 691
several tables, example, 226
syntax, 691
Update method (ADO.NET), 629
updateXxx methods (Java), 540
updating MySQL, 36
upload
files, 404
to ISP, 369
with phpMyAdmin, 113
upload_max_file_size (PHP), 408
UPPER, 694
URL, 414
urlencode (PHP), 428
Usage privileges, 275
USE, 691
use method (C++), 572
user
interface, 86
forum example (PHP), 467
mysql.user table, 279
USER, 701
user variable (@), 651, 684
user-defined function, 662
user name, 263
users
creating new, 272
registering new, 291
user table, 280
user table (access protection), 279, 778
USING
DELETE, 666


Index_V

Team LiB

Index

V

VALUES, 671
VARCHAR, 142
variable, 207, 651, 684
VARIABLES, 690
VB, xviii
VB.NET, 616
VBA, xviii, 592
VERSION, 701
version numbers, 11
views, 10
virtual directory, 42
Visual Basic, 592
.NET, 616
Unicode, 601, 605


Index_W

Team LiB

Index

W

wasNull method (Java), 537
WBK (well-known binary representations), 774
web.xml file, 551
WEEK, 697
WEEKDAY, 697
WHERE, 183
Windows service (MySQL), 24, 28
WinMySQLadmin, 93
installation, 25
security risks, 270
WITH ROLLUP, 777
WRITE LOCK, 676


Index_X

Team LiB

Index

X

X509
GRANT, 669
identification, 282
XML
mysqldump, 726
XML export
mysql, 331
mysqldump, 329
XOR, 647
$_XXX fields (PHP), 376


Index_Y

Team LiB

Index

Y

YEAR, 697


Index_Z

Team LiB

Index

Z

ZEROFILL, 655


List of Figures

Chapter 2: The Test Environment

Figure 2-1: WinMySQLadmin
Figure 2-2: Starting and Stopping MySQL as a service
Figure 2-3: Testing the MySQL installation under Linux
Figure 2-4: Apache basic configuration
Figure 2-5: Apache's default start page
Figure 2-6: Access to a web directory protected by .htaccess
Figure 2-7: PHP 4 runs!
Figure 2-8: PHP can access MySQL.
Figure 2-9: phpMyAdmin
Figure 2-10: CGI test page

Chapter 3: Introductory Example (An Opinion Poll)

Figure 3-1: The questionnaire
Figure 3-2: Results of the survey
Figure 3-3: The MySQL monitor

Chapter 4: User Interfaces

Figure 4-1: Setting the Windows system variables
Figure 4-2: Using the MySQL monitor under Linux in an xterm window
Figure 4-3: Configuration of the MySQL monitor under Windows
Figure 4-4: WinMySQLadmin as an icon
Figure 4-5: Setting my.ini in WinMySQLadmin
Figure 4-6: Setting up a new server
Figure 4-7: Database administration with the MySQL Control Center
Figure 4-8: Executing SQL commands
Figure 4-9: User administration with the MySQL Control Center.
Figure 4-10: Start page of phpMyAdmin
Figure 4-11: Database view of phpMyAdmin
Figure 4-12: php table view
Figure 4-13: Login dialog for http authentication
Figure 4-14: Login dialog for cookie authentication
Figure 4-15: Design of the table titles in phpMyAdmin
Figure 4-16: Table view with many options for changing the design
Figure 4-17: Adding a data record
Figure 4-18: Adding a password
Figure 4-19: Page-by-page representation of data records
Figure 4-20: Executing a database backup
Figure 4-21: Text importation of table data
Figure 4-22: References by the table titles to other tables in the database mylibrary
Figure 4-23: Listboxes simplify the input of foreign key fields.
Figure 4-24: Setting the layout of the pdf table schema
Figure 4-25: The resulting structure as a pdf document

Chapter 5: Database Design

Figure 5-1: The structure of the database books
Figure 5-2: Structure of the mylibrary database
Figure 5-3: Example data for the categories table
Figure 5-4: Query result in search for higher-ranking categories
Figure 5-5: Structure of the myforum database
Figure 5-6: A discussion thread

Chapter 6: An Introduction to SQL

Figure 6-1: First SQL experiments in MySQL monitor mysql

Chapter 10: Administration and Server Configuration

Figure 10-1: A table generated with mysql

Chapter 11: PHP — Fundamentals and Programming Techniques

Figure 11-1: A simple PHP example
Figure 11-2: The result of a simple query displayed as a table
Figure 11-3: Testing magic quotes
Figure 11-4: Storing and displaying images

Chapter 12: PHP — Library Management

Figure 12-1: The search form
Figure 12-2: Results of a search for book titles beginning with the letter M
Figure 12-3: Results of a search for authors whose last name begins with K
Figure 12-4: Links to additional pages with search results
Figure 12-5: Simple book title input
Figure 12-6: Input of a new book title (phase 1)
Figure 12-7: Checking the input (phase 2)
Figure 12-8: Listbox for languages
Figure 12-9: Hierarchical listbox for selecting the book category
Figure 12-10: Selection or new input of a publisher name in the second input phase
Figure 12-11: Select or delete a category
Figure 12-12: Adding the category Visual Basic in the group All books | Computer books | Programming

Chapter 13: PHP — Discussion Forum

Figure 13-1: Login dialog for the myforum discussion forum
Figure 13-2: Establishing a new myforum account
Figure 13-3: List of all discussion forums
Figure 13-4: List of all postings to testforum
Figure 13-5: A single posting
Figure 13-6: An entire discussion thread
Figure 13-7: Form for composing a message
Figure 13-8: Hierarchical representation of three threads (ordered by flow of the discussion)
Figure 13-9: Flat representation of the ten most recent messages from Figure 13-8 (ordered by time ofcreation)
Figure 13-10: Hierarchical representation of the most recent ten messages (ordered according to the discussion flow)
Figure 13-11: Representation of messages via forumread.php

Chapter 14: Perl

Figure 14-1: Searching for book titles
Figure 14-2: Simple book title input

Chapter 15: Java, JSP, JDBC

Figure 15-1: A first JSP example
Figure 15-2: JSP example with traditional Java code

Chapter 17: Visual Basic, C#, ODBC

Figure 17-1: Determining the Connector/ODBC version number
Figure 17-2: Basic Connector/ODBC setting
Figure 17-3: Connector/ODBC Options
Figure 17-4: Selection of the tables to be imported into Access
Figure 17-5: Excel pivot table
Figure 17-6: Selecting the desired columns in MS Query
Figure 17-7: MS Query in development mode
Figure 17-8: DataEnvironment dialog for setting the connection properties
Figure 17-9: Visual Basic program to test Recordset properties
Figure 17-10: Visual Basic example program with the MSHFlexGrid for representing data from the mylibrary database
Figure 17-11: Visual Basic example program for changing the titles table
Figure 17-12: Visual Basic example program for inserting a new book title
Figure 17-13: The photograph is loaded from a BLOB field and displayed
Figure 17-14: ADO.NET example program
Figure 17-15: Form for storing a new book title
Figure 17-16: Graphical user interface for mssql2mysql


List of Tables

Chapter 1: What Is MySQL?

Table 1-1: MySQL functions past and future

Chapter 4: User Interfaces

Table 4-1: mysql commands in interactive mode

Chapter 5: Database Design

Table 5-1: A book database: first attempt.
Table 5-2: A book database: first normal form
Table 5-3: title table: second normal form
Table 5-4: author table: second normal form
Table 5-5: title table: second normal form
Table 5-6: author table: second normal form
Table 5-7: rel_title_author table: second normal form
Table 5-8: title table: third normal form
Table 5-9: publisher table: third normal form
Table 5-10: Database representation of the hierarchy in Figure 5-3
Table 5-11: Query result in the mylibrary database
Table 5-12: categories table with hierNr column
Table 5-13: Some entries in the rel_cat_parent table
Table 5-14: messages table with records of a discussion thread
Table 5-15: messages table with records of a discussion thread

Chapter 7: SQL Recipes

Table 7-1: Boolean search expressions (IN BOOLEAN MODE)

Chapter 8: InnoDB Tables and Transactions

Table 8-1: Coordinating transactions

Chapter 9: Security

Table 9-1: MySQL privileges
Table 9-2: Structure of the user table
Table 9-3: Default setting for the user table under Unix/Linux
Table 9-4: Default setting for the user table under Windows
Table 9-5: Structure of the db table
Table 9-6: Default setting of the db table under Unix/Linux
Table 9-7: Structure of the host table
Table 9-8: Structure of the tables_priv table
Table 9-9: Structure of the columns_priv table

Chapter 10: Administration and Server Configuration

Table 10-1: Query cache status variables

Chapter 11: PHP — Fundamentals and Programming Techniques

Table 11-1: Superglobal fields (available since PHP 4.1)

Chapter 15: Java, JSP, JDBC

Table 15-1: Components of the connection URL
Table 15-2: Optional parameters in the connection URL (with default settings)

Team LiB
Previous Section