|
Determine overall data collection approach
Mapping out the “big picture” provides a framework that
identifies which entities—such as recyclers, government agencies, and
businesses—need to be surveyed. The
number of generators the jurisdiction will need to survey depends upon the size
of the jurisdiction. For example, if the
jurisdiction is small, it may be feasible to survey all of the non-residential
generators in the community.
Establish when surveying generators will require use of extrapolation
methods
Depending on the size of the
community, the number and types of generators in the community, and the amount
of previously collected information, all or some of the potential data sources
need to be included in the diversion study. For some communities, surveying the largest generators may be the best
approach.
For example, the City of
Monterey had 3,500 total non-residential generators. After ordering the non-residential generators list by number of
employees, the city began to survey the largest non-residential generators
first. The generators became
significantly smaller after about 150 generators were surveyed. The city found that by surveying the largest
generators it captured significant diversion tonnage. With limited staffing resources, it proved more cost-effective to
survey only the top 150 generators.
Also, depending on the study
objectives, a jurisdiction may elect to conduct a comprehensive study to
quantify all measurable diverted materials, extrapolate from a sample of
material generators, or only examine specific programs or business sectors. With a well-designed strategy, it is usually
possible to get adequate data with less than a “100 percent complete” survey of
all targeted programs. The following
generalizations are typical for many jurisdictions within each size grouping;
however, exceptions do exist.
- Small jurisdictions have less complex
waste management systems.
- They are able to design less intensive
studies while still obtaining good, accurate data.
- They have just a few large generators of
materials, a single hauler system, and only a few diversion programs. They may be able to do a quality diversion
study with just a small number of phone calls.
- Chances are good that a smaller
jurisdiction already knows about the major governmental, commercial, and
industrial material generators, and they may operate the majority of the
diversion programs themselves.
- Documenting the known existing recycling
efforts and a limited survey of private source reduction activities may be all
that is required. There is usually no
need to develop a sampling strategy beyond setting a cost-effectiveness or
percentage of significance, and a cut-off point for collecting the data.
Medium-sized jurisdictions
- If the community is primarily residential
and has only a few large commercial and industrial generators, a sample-based
estimate and a size-based business audit may be relatively quick and easy to
do.
- As economic complexity increases, the data
collection problems begin to resemble those of a large city.
Large jurisdictions
- Large jurisdictions face a unique array of
potential issues, generally without time or resources to complete a survey of
all diversion data sources.
- If the jurisdiction has a large number of
commercial and industrial facilities, it is recommended that the largest (e.g.,
as previously determined by number of employees or waste disposal tonnage) of
the businesses be targeted first. This
is applicable to partial enumeration, non-extrapolative measurements.
- For larger cities, the majority of the
waste volume is usually produced by an identifiable percentage of
non-residential generators. While
assessing the characteristics of the non-residential community, a split is
often noticeable between the large generators and smaller generators.
Larger jurisdictions may not
have the resources to survey all of their largest businesses. In this situation, jurisdictions may choose
to randomly sample their entire business sector.
Decide appropriate sampling method
Once you have compiled a list of
all of the residential and/or non-residential sector entities that will be
surveyed, enter the data into a spreadsheet or database. The data should include contact information,
location, and number of employees or residents, if appropriate. For the non-residential sector, the entities
can be ranked according to number of employees. If actual diversion data for each business is available, it could
be used instead of ranking by employees.
Statistical
theory states that if a proper sample is drawn from a large population, all the
characteristics of the population will be reasonably represented in the
sample. Medium and large jurisdictions
will find it cost-effective to apply this mathematical technique by selecting a
representative sample of sites to survey and extrapolating the diversion rate
for the entire surveyed population.
There are
four basic types of random statistical sampling using a list of all possible
sites (such as a property tax list of all residences or a list of all business
permits), which include:
Simple random sample
- A simple random sample is obtained by
choosing elementary units in such a way that each unit in the population has an
equal chance of being selected. A
simple random sample is free from sampling bias. However, using a random number table to choose the elementary
units can be cumbersome. If the sample
is to be collected by a person untrained in statistics, then instructions may
be misinterpreted and selections may be made improperly. Instead of using a list of random numbers,
data collection can be simplified by selecting, say, every 10th or 100th unit
after the first unit has been chosen randomly, as discussed below. Such a
procedure is called systematic random sampling.
Systematic random sample
- A process whereby a random sample is
obtained by selecting one unit on a random basis and choosing additional
elementary units at evenly spaced intervals until the desired number of units
is obtained. For example, there are 100
students in your class. You want a sample of 20 from these 100 and you have
their names listed on a piece of paper in alphabetical order. If you choose to
use systematic random sampling, divide 100 by 20, and you will get 5. Randomly
select any number between 1 and 5. Suppose the number you have picked is 4,
that will be your starting number. So student number 4 has been selected. From
there you will select every fifth name until you reach the last one, number
100. You will end up with 20 randomly selected students.
Stratified sample
- This method requires that the sample be
obtained by independently selecting separate simple random samples from two or
more relevant population strata. A
population can be divided into different groups (strata) based on some
characteristic or variable such as number of employees, income, or
education. For example, waste generators
with 100 or more employees will be in group A, waste generators with between 50
and 99 employees will be in group B, and waste generators with between 1 and 49
employees will be in group C. These groups are referred to as strata. You can then randomly select from each
stratum an appropriate number of units (sample size).
One method of deciding how many samples to draw from each group
or strata is to select based on the representative proportion of each group to
the whole population. For instance, if
the total population of generators is 1,000, and group A has 300 generators,
while group B has 150, and group C has 550, you may decide you will take 30
percent from group A, 15 percent from group B, and 55 percent from group C. Proportional sampling from each strata is
only appropriate if the individual strata are homogeneous with regard to the
population characteristics of interest. In other words, when sampling for proportion, if your characteristic of
interest is grasscycling, then each member of each strata must have the
capability to grasscycle.
Cluster sample
- Cluster samples are obtained by selecting
clusters from the population and then sampling each cluster using a simple
random sampling approach. Using this
method, a sample comprises a “census” of each random cluster selected. A cluster is often chosen using some type of
natural or artificial spatial attribute, like political boundaries, parcel
zoning, or even entire jurisdictions.
As an example, suppose you decide all the elementary schools in Los
Angeles are clusters. You want 20 schools selected. You can use simple or systematic random sampling to select the
schools, and then every school selected becomes a cluster. If your interest is to interview teachers on
their opinion of some new solid waste education program that has been
introduced, then all the teachers in a cluster must be interviewed. It’s worth
noting that while cluster sampling can be very economical, it is very
susceptible to sampling bias. In the
above case, you are likely to get similar responses from teachers in one school
due to the fact that they interact with one another.
Note: When using cluster and stratified sampling
in medium or large jurisdictions, groupings should be made using either the
four- or six-digit SIC code. This level
may not be useful in smaller communities. Jurisdictions should work with a statistician to determine groupings.
The key to
accuracy in random sample selection is to select the appropriate sampling
methodology and then ensure that each potential participant has an equal chance
of being selected from a complete list of potential sites. To ensure equality, the choice of possible
random numbers should be limited, depending on the number of possible sites, so
that at least three complete passes are made through the entire list of
candidates without replacement.
The potential errors in statistical
sampling are numerous and difficult to identify and control. Unless the
jurisdiction has had successful previous experience in designing and executing
a statistical sampling plan, it would be wise to seek assistance. The final report should include a detailed
description of all survey methods and statistical analyses, including adequate
documentation of the survey design, implementation, and analysis used. See Appendix
J for determining number
of samples needed.
Recommended methods for sample
allocation:
- A jurisdiction should survey all very
large businesses that contribute a significant portion of the waste
stream. Although these businesses may not
have been randomly selected, the waste stream would not be correctly
represented without them. Allocate the
remaining samples by subtracting the number of large businesses surveyed from
the total number of businesses. Use the
remaining number of businesses and the table in Appendix K to determine the
survey sample size for the remaining businesses.
- Do not extrapolate diversion tonnage from
extremely large businesses, or those with atypical diversion practices. Add the actual diversion tonnage from all
very large businesses to the diversion tonnage extrapolated from the randomly
selected businesses.
- Do not extrapolate restricted waste
diversion tonnage.
- If a jurisdiction does not have a
significant “large business stratum” so that the number of samples allocated to
the large stratum exceeds the actual number of large businesses, sample all of
the businesses in the large stratum and randomly allocate the balance of the
samples in the smaller business stratum.
To assist jurisdictions with
determining sample size for residential and non-residential surveys, see
Appendix J. If you need assistance with
determining sample size for a residential or non-residential survey of waste
diversion, please contact your CIWMB
local assistance staff contact at
(916) 341-6199.
Develop data collection tools
The next step is determining the
data collection strategy. Gathering
information is expensive in terms of personnel, time, and budget
resources. The jurisdiction should
always strive to obtain the needed diversion data with methods that require the
least effort and minimal cost to obtain, but obtain the appropriate level of
accuracy. If additional tonnage needs
to be documented, sources and methods should be selected in the order of least
to greatest effort and cost.
The appropriateness of each
survey type depends on factors such as the amount and detail of information to
be collected.
- Mail surveys are often used to collect
data on residential diversion activities such as composting or grasscycling.
- Telephone surveys are sometimes used to
collect data on non-residential diversion activities when the generators are
small or have simple sets of activities.
- On-site surveys are preferable for large
generators and those generators with detailed sets of diversion activities.
Mail surveys
Pro:
- Can provide an effective means of
gathering residential data (e.g., grasscycling).
- Least expensive method of gathering
diversion information.
- Most appropriate for surveying large
populations using simple and easy-to-answer questions.
Con:
- Usually receive the lowest rate of
participation.
- Require a long processing cycle before
results are available.
- Problems with poorly designed forms or
misunderstood questions are often not discovered until after the survey forms
have been completed and returned.
- May require a telephone call follow-up to
remind the participant to complete the survey and for data clarification and/or
verification. Otherwise, many
households and most businesses will simply discard the survey forms.
- Survey questions can be easily designed to
bias the outcome.
- Self-selection bias on the part of
respondents.
Telephone surveys
Pro:
- Allow for more complex questions.
- Provide information quickly.
- If the survey method is defective and does
not elicit useful data, the questions or format can be corrected early in the
collection process.
- Give more flexibility in following up with
answering questions.
Con:
- Almost twice as expensive to do as mail
surveys.
- Can’t confirm details.
- Questions that require detailed research
by the participant will require one or more follow-up calls.
- Can lead to bias from the surveyor.
- Self-selection bias on the part of
respondents.
- Can be difficult to administer in a
multilingual community.
On-site surveys
Pro:
- Highest response rates
- Allow the largest number of questions.
- Provide for a visualization of actual
diversion activities.
- Provide an opportunity to give hands-on
technical assistance to the waste generator.
Con:
- More expensive to perform than mail and
telephone surveys.
- Can lead to a bias from surveyor.
- Self-selection bias on the part of
respondents unless participation is mandatory.
Note: To facilitate a discussion with the
generator, it can be extremely helpful to have a contact at the business or
government agency responsible for the waste management activities. If the jurisdiction has a franchise hauler,
the hauler often has contact information for each business and government
agency.
Regardless of the survey method
chosen, great care needs to be taken in the design of the survey questions. Survey participants will sometimes
misunderstand questions or overestimate their frequency and/or tonnage
contributions to programs. It is also
important to remember that the variable definitions used by the different
suppliers of information in the study are a large potential source of data
error.
All survey forms and letters
must simply and clearly identify the details of the desired diversion
information. Questionnaire bias can be
checked by asking for the same information in different ways in different
places on the survey form, or by testing survey results against reality. This requires conducting follow-up with
generators that were surveyed.
The CIWMB has developed a
general on-site diversion study survey form (see Appendix
A, MS Word 2000, 28 kb), which can be adapted
for a jurisdiction’s diversion study. Other survey forms have been developed by jurisdictions that have
conducted diversion studies; you may wish to contact them to obtain samples of
other survey forms.
Response rate
Participation in a diversion survey
can result in a response rate range of 20-30 percent for mail surveys to 70-80 percent for on-site surveys.
The
response rate depends on the clarity and ease of use of the survey form, the
nature of the information requested, the participant’s interest in the topic,
and the amount of staff follow-up on unanswered forms.
- Over sampling may be necessary because not
all homes and businesses will participate in a survey. Appendix
J includes a table to assist in
determining sample size. The number of
samples listed is the number of completed samples required.
- A small pretest sample should indicate the
response rate for a larger survey. Each
attempt to follow up on non-respondents will usually yield about half the
previous participation rate, so it is generally not cost-effective to make more
than two additional contacts for the information.
- Ensuring confidentiality of the data may
sometimes increase response rates for the data provided, but the additional
procedures needed to guarantee confidentiality will increase the cost and
complexity of the project.
- Requesting only masked (customer’s name
removed) or summary information for some waste haulers can be an effective
means of including confidential data in the survey process.
The jurisdiction must take into
account all restricted wastes outlined in statute (agricultural wastes, inert
solids, scrap metals, and white goods). Most of the guidelines about what counts (or doesn’t) toward diversion
of solid waste are defined in statute, in PRC sections 41780 through 41786, with further
guidance in regulations (Division 30, “Waste Management;” Part 2, “Integrated Waste Management Plans;”
Chapter 6, “Planning Requirements;” Article 1, “Waste Diversion”). The criteria
for counting something toward diversion in a 1990 or 1991 base year also apply
when establishing a new base year (e.g., 1999).
In addition, all waste types for
which a jurisdiction claims diversion credit must meet the “normally disposed”
criterion defined in Title 14, California Code of Regulations (14 CCR), section
18720(a)(44). The aforementioned
statute and regulations have been compiled into a concise document “What Counts
Toward Diversion” (see Appendix H). The
restricted waste criteria are in addition to the “normally disposed” criterion
that applies to all waste types counted toward diversion. In addition, restricted waste diversion
tonnage cannot be extrapolated.
Before addressing restricted
wastes, the jurisdiction or consultant should read the restricted waste section
of Appendix H (“What Counts Toward Diversion”), which describes the restricted waste criteria set forth in
statute. Restricted waste analysis can
be complex, so if you have any questions, contact your local assistance staff representative.
Next section | Table of contents
|
